From patchwork Thu Feb 23 03:04:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13149801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2CA8C636D6 for ; Thu, 23 Feb 2023 03:05:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B02C6B0078; Wed, 22 Feb 2023 22:05:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 75F786B007D; Wed, 22 Feb 2023 22:05:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 627756B0080; Wed, 22 Feb 2023 22:05:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 548AB6B0078 for ; Wed, 22 Feb 2023 22:05:04 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 10C20A0F1B for ; Thu, 23 Feb 2023 03:05:04 +0000 (UTC) X-FDA: 80497064928.23.A35DF5A Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf23.hostedemail.com (Postfix) with ESMTP id 3C92C140002 for ; Thu, 23 Feb 2023 03:05:02 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=CmpbZQ5m; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677121502; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7w1cKhAAphXJ2vOUap7qolI6gnUUf7d9gByR138SJog=; b=c3us1dyNXpW7uEeokLRKD70JsIQGnb2qTpP2mkMLYJp/E4GtPiGmsm0lRjTzO1c2LYw0Lp VuCuCyU0kJmsOHrm6Isj3MVC03H4fk8dygTfK4jIhFlK7AjksHbz8Vrt17zymXjGYpH5ex O4Y07o93AicDhXEeAq+i2pX79DEIqRU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=CmpbZQ5m; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677121502; a=rsa-sha256; cv=none; b=adgEN21OGBBAbyCRNq6g9glfgNNstV75G2j4BY+qT2ARTiJEi0AAYgSSWxj9WmdQar3X0u GCqohycqgcsTvlSQLvfgI0iexMgQoowuHEwwrYOXTCMKT5A0thGJZnOPfdpdb81fGWupxM cwEr0zua6EE7VFIKOpKZHKzk7Xkk3/Y= Received: by mail-pj1-f43.google.com with SMTP id cp7-20020a17090afb8700b0023756229427so2350312pjb.1 for ; Wed, 22 Feb 2023 19:05:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7w1cKhAAphXJ2vOUap7qolI6gnUUf7d9gByR138SJog=; b=CmpbZQ5mXsvkYH52sH4fUdQAKboV5mAC+PL5L2EQo9+52dN8FTsKfuwkHN31kEURgt zJmYtdNdQZoFlGbZMbVJxlfCAiCY1teEF6fHg9QB2ulPwRSIVYrc1aMPXJGQEnc5XvNi xpv0MPmkAscsnxtGgUib+qZbsAdfI51mc7cf4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7w1cKhAAphXJ2vOUap7qolI6gnUUf7d9gByR138SJog=; b=j0/+GcEVLbYJZkKAgMvZ1gNYk/xrGf92tOG0KEPlVAYB4TzlhA3Fshqsq4LWDED24Z 5JRXNpHyMpR8jrEjOePwU2uKrFTabzl3kTBUwPzWxRB2D88TJzmL39QINAAYPBbpQUIZ UQDLp08iVSpHtzyOszSWQVLgvNTDfjESEuHajkRQsSiT2h4+5Zy4mYf3fsKmYJbTtPFL yimd+9ofJQfjiOT7uc+seOAAQs7/rVJqi40LSMpz7ALb12kv76rhmSHgBdyg1vpNmwC8 zQh56rWl2AkEjbD/UH6Fq27yCSe51liMiikk0NQ9yYbrQWgYkYmgsoDxcPrbhgG7NjDD VdoQ== X-Gm-Message-State: AO0yUKXBRobeRU6d3jIXFRo3kBCQ0U05H1bIEredYM7uLht3RbDdvGXt 7OXJVRREweLKXrmD71DH2ae/3A== X-Google-Smtp-Source: AK7set/dsMOb60qr+5+6Ogrh8xQz946tMFAG3MvgpCazePEMrr3hTDtUIcO7lCYLWoHiifUoUwS7ew== X-Received: by 2002:a17:902:7243:b0:19c:bcbc:a3dc with SMTP id c3-20020a170902724300b0019cbcbca3dcmr685857pll.25.1677121501195; Wed, 22 Feb 2023 19:05:01 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6de2:9e85:b508:57b8]) by smtp.gmail.com with ESMTPSA id jl21-20020a170903135500b0019926c77577sm608520plb.90.2023.02.22.19.04.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 19:05:00 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv2 1/6] zsmalloc: remove insert_zspage() ->inuse optimization Date: Thu, 23 Feb 2023 12:04:46 +0900 Message-Id: <20230223030451.543162-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog In-Reply-To: <20230223030451.543162-1-senozhatsky@chromium.org> References: <20230223030451.543162-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3C92C140002 X-Stat-Signature: 675mxdooxmj9p9yy88xqxum816w9gq9j X-HE-Tag: 1677121502-622565 X-HE-Meta: U2FsdGVkX18FeL/WDEqdKHp1+75TNFCXtvziskxIlaLy8XS8oNQoy3pNm2u0DREzdh/mMKkupL+zFRBrBl0KRdeno5s5nFMIOHjUgzRh+nUXqATvJydTfFEa+FhuabhvJS+x10ozz2DCh/nT42OMNitb875trFlvJ7teGBgMLZc8Wd/HpXLJ4S1rGYVW+jXG+r7iLAjLsYH7nWpNS4pLoIo+6DbNbhkSE7/U8LXEvQ1AruzjVw9vAIn8m3MrSJHo+yVXfRZve7Unkv0ZMOrQtaclr3LJkfMfy79Ag0pjoOQx6yhuV8bs5AeMEDdsSG8poKrVKop15gWOcTzBYwmaYukkQwcy0pBdM3ogJi5F9GF82D4esL8NiC8aQ1eo6xmN50ipmdAM2zZjB5tCBiIdK9/7ved0JTTc8Aeqc4XpfRxRT+xrqB6+SIrViRuGI07Wsu3yNlHwL1qq4wMBCBUSQN1IynKSg0wq7nwNuu1rsN9QaOXdOh0i1i7hfPgTmKfD+i0NiiZiQerAhXejJ0Ou8aGThl0m5wQuje2m/hJehG08v/7zaMGa878L+t1dPz57jccJ974n5Z7VSi/Pd1i2fBxmN0Hrztq13NF0vUdJVu1GNqIlrK6EP4YeqQGcH8LV3/VzgpWvO13lXK+aP8FxCrXrtiEGSAHExBQ8/mWSxbt3mZN8J1tufOcthT6Kz3/t11eQrvea1c6324MbHVbc3swpGyWX021fOFNFco49x3rjeFy+yqN8Ln54ekV/6bbaBLLUEPk7JAXbYNvu/bA0Q4Fw6bCp2G6h5K+0Q7x6AxwEbsqVkybNm0A5gRPuEu0cod4KQR2LdjJ9vhZgMzqHyAWyaGzcphYv59MejG1ZC87QoxftpxD53Q8DoaMLkrWX//8YF3r2jvCbOR29wMgK0SpCpSVbwt63nXm7oei7mRSNwGBtJqWMn0Wsnl8PF/aW1gBpBU4HhT/qZJQxbdW 1Pr/A2xB Bz1GYJOhkDc5YZmR51oE6fHnzlZXQO35ili5VccIGTYFL77nA9OjvjRx0BDTQd817lhygTzWk9d46lhDreKyZsJV5pXAYq5lddg/AHYXO1EoL4AjL4xBdmBGc70VgIkHpNt23gJEcGZCd1VleMKDR7KhWTtbYeWDnOmlzFNF/KpsPOzMJ0/XOa8Z3+fBdYSWX886Li7SYKzlcQbF52T0pdKysnBrQV2iFuZRG7ApeIUSf7PqLxLblBMQDZjB61GnaSop8FueK2dOT/E/ZnBHFa6qNasgOMI8jjXickA2gGvWxv3uCrbafc7WE75oTPFCZHFIAZLnCT4z9sqFC/ekOdbtRN1BZsmGAlyHyZag74/+Gput2YhnjVsbYI9KEI7j194tsX+dkyySRvZfbjZEyeU5KvO6ZQkKEZW27WGTf6zaVKM4BSnBpY837SAADHrG0fLWQXSsghkTVC4whl6CIEafiCZI92y2GAqys X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This optimization has no effect. It only ensures that when a page was added to its corresponding fullness list, its "inuse" counter was higher or lower than the "inuse" counter of the page at the head of the list. The intention was to keep busy pages at the head, so they could be filled up and moved to the ZS_FULL fullness group more quickly. However, this doesn't work as the "inuse" counter of a page can be modified by obj_free() but the page may still belong to the same fullness list. So, fix_fullness_group() won't change the page's position in relation to the head's "inuse" counter, leading to a largely random order of pages within the fullness list. For instance, consider a printout of the "inuse" counters of the first 10 pages in a class that holds 93 objects per zspage: ZS_ALMOST_EMPTY: 36 67 68 64 35 54 63 52 As we can see the page with the lowest "inuse" counter is actually the head of the fullness list. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 29 ++++++++--------------------- 1 file changed, 8 insertions(+), 21 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 3aed46ab7e6c..b57a89ed6f30 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -753,37 +753,24 @@ static enum fullness_group get_fullness_group(struct size_class *class, } /* - * Each size class maintains various freelists and zspages are assigned - * to one of these freelists based on the number of live objects they - * have. This functions inserts the given zspage into the freelist - * identified by . + * This function adds the given zspage to the fullness list identified + * by . */ static void insert_zspage(struct size_class *class, - struct zspage *zspage, - enum fullness_group fullness) + struct zspage *zspage, + enum fullness_group fullness) { - struct zspage *head; - class_stat_inc(class, fullness, 1); - head = list_first_entry_or_null(&class->fullness_list[fullness], - struct zspage, list); - /* - * We want to see more ZS_FULL pages and less almost empty/full. - * Put pages with higher ->inuse first. - */ - if (head && get_zspage_inuse(zspage) < get_zspage_inuse(head)) - list_add(&zspage->list, &head->list); - else - list_add(&zspage->list, &class->fullness_list[fullness]); + list_add(&zspage->list, &class->fullness_list[fullness]); } /* - * This function removes the given zspage from the freelist identified + * This function removes the given zspage from the fullness list identified * by . */ static void remove_zspage(struct size_class *class, - struct zspage *zspage, - enum fullness_group fullness) + struct zspage *zspage, + enum fullness_group fullness) { VM_BUG_ON(list_empty(&class->fullness_list[fullness])); From patchwork Thu Feb 23 03:04:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13149802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2CEAC64EC7 for ; Thu, 23 Feb 2023 03:05:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 31B356B007D; Wed, 22 Feb 2023 22:05:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CC4D6B0080; Wed, 22 Feb 2023 22:05:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16CAC6B0082; Wed, 22 Feb 2023 22:05:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0ABD86B007D for ; Wed, 22 Feb 2023 22:05:07 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CE9AE81027 for ; Thu, 23 Feb 2023 03:05:06 +0000 (UTC) X-FDA: 80497065012.06.8732B75 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf27.hostedemail.com (Postfix) with ESMTP id 06CB34000D for ; Thu, 23 Feb 2023 03:05:04 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="PVE/woFY"; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677121505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I+a7RvRhurz1Ai+VZTI1SaXOAjw7CakUvWehQGGNcJ0=; b=z2fjYiKgsxHBWHbqL6RAY8VpfFLA/GfQjEHChIbYLRSgsYzTKUyVmGf90VHmQku+b594Jp Iw3GKy+nWiw+241G9x9/UuO+4IFC7HmqNK0K/SjRfsUOSAdzDby5lpqq3nKJpcPa/YLHSg mkaXbeuYSFtz9tTDWczThRL7u0lCl9Q= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="PVE/woFY"; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677121505; a=rsa-sha256; cv=none; b=Qw2OJFNGdxbpSPtZ6jPtQjRnXcrP3TBVz8r7IfIPu74iZ1vGi7iLL7E5jSHRT0t7WRuyZX R0fSzqLMLa3V169a5Ssew8+P2ODiWTJVek5Xye0b/sON/v2BlaBqbTMB1BsUfQB8cWHnEB q0vQdjxuBJm6IQHYvG1vbZ5rHEMfeqY= Received: by mail-pl1-f179.google.com with SMTP id h14so11397687plf.10 for ; Wed, 22 Feb 2023 19:05:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I+a7RvRhurz1Ai+VZTI1SaXOAjw7CakUvWehQGGNcJ0=; b=PVE/woFYX/y2Nv/sSWN/ThQ7WpqOXClosC23oGEVbNLw+KQEHwBO6RQseLQLRiTTO8 rQ3VjL8xKc7Wgb/Jl357h+iW+SZvTOhE+otycFTz0/T665RZa055Y/XJ5GzLILgml5tC 2n9hS+yqHu+gRYPQ1oJQZHmz4yUqpgjh5Y0UU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I+a7RvRhurz1Ai+VZTI1SaXOAjw7CakUvWehQGGNcJ0=; b=K/NwU8jUDYrIwGyvrm2JIK4+8YFmcze4oxL1HLZPsx1WpNmg/yhFNGVAagCieFarP2 vcQZ+/dkBZzVVDOJBG2dcnOJ+aDTsz18ijj3mf9RAUUYdp5WKb7nXCqDCaFYc2nQP5Uy 44LaKxluCQnJaCHEbvUjUd2MTYSPaJeD0DRE+FbQn9V4Skua7lbUAgsKlPRIc7EYJtBI AuY6Vg7wDweifIDl6VY/6UiQJetx9ED5xgt/JWe14LYvBAEt+KQCN0m+hABTp8BKOxPj olAXXbLXaKVmT1pHpJ92fPRf4qQst+cteLm5cHa+hxyZeV+CpjFeg9a0SkNYvhfgXUQ9 6rAg== X-Gm-Message-State: AO0yUKW0fMoO6qR6nD5RsaUz6z+AgS4aZ3QZewgm1tsXc3T30zk6Yi7m mnwsUGck1APvZH8sCJR4LTfidw== X-Google-Smtp-Source: AK7set9iWFHlpCYsAD2np6Tp3JHdZO5a9W4ONaB5Ya2yJKg+38hrP2KF/ufMSur/h6V0NBGOjezMpQ== X-Received: by 2002:a17:902:ecd0:b0:19b:2332:18cb with SMTP id a16-20020a170902ecd000b0019b233218cbmr12943938plh.1.1677121503834; Wed, 22 Feb 2023 19:05:03 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6de2:9e85:b508:57b8]) by smtp.gmail.com with ESMTPSA id jl21-20020a170903135500b0019926c77577sm608520plb.90.2023.02.22.19.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 19:05:03 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv2 2/6] zsmalloc: remove stat and fullness enums Date: Thu, 23 Feb 2023 12:04:47 +0900 Message-Id: <20230223030451.543162-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog In-Reply-To: <20230223030451.543162-1-senozhatsky@chromium.org> References: <20230223030451.543162-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 06CB34000D X-Stat-Signature: toydxr1xqyx6wk18o8uqigxrcfztia9z X-Rspam-User: X-HE-Tag: 1677121504-486287 X-HE-Meta: U2FsdGVkX18fKUQIDkMlRRwJzjGCv04vaGxDKkHSM0kHXx70WRhg91fSIl/u4Y8oNz3QcmzdEDfjaSbgb4KrepKIpwg1RyOVMV5YeDI9y75VoP/9rQI4Al/36xtUhtp3LGB0TL2dH5uhC9gu09EBkoWVJxOEsJFf3isT3a2ynzlXQ26nz9f+5k1qmV/qM1JY2lGsmkONxHBBLhKLo1CDh5tPxMmhakWhoSUxOteYGsTWCPQqnISknFtzTvny1gBQGTU+tEX6HqvyjTIjDeiqFVJGTvQx2VwJUqGxkbqfp1DVLSAHfKzEsSF8fsn9IeCR55uDh9sG5w683/Cop63bs7cDDp4dKwVsMdiJig9/X5gplfbHVrZ7xzdVez95SPVISpTlQWcYzl9j2aYHUvE4bcg1jPpKYBjjSwi7eWVc2stf8mAG1R+xvMk/pMOK02secBVyAZ5BbkZvwB9nwmtX/b8bcTXlVm59cg3Zc1aVKojOiSNWanwURliml69JToZHIX6Y3f3KECeQnqzdL6gqVF6ZnScBpMxvaBAvxl9YjtsHcAUM5XnqmVZ0LqcJEqW5TtIuco2NCQ/CTQNbW1yPQxUU9WIKztr6zF6gx/QEm/K7F/yWUpw0ZXCtvL/8u+H/aCzv0KL/ltekLf0BeBiRHZem8fl3WlV842sOamWdIIVjX+4XVSvSRXUEJNhvNJGZkjjCQ1CA7ar5FbfquvKIiGno22fsE0y4ZpunDUoNGc5jryozdnQ8W5oau+xa3Cauy+TyioY5tid3ZMuE+ZnrzzEAzatOnZ4zUaNcvRlTlWmwMmH+C18K0/c0z9hu+AS25JbMFjsonxD8aGWMb6w+cnQf+IJSxGoKFxF9bV1mbiZOJauaY0gZRgnLekZ3tomtlriI+C2wj9AJjK6zi4bw3JhE+7i7NIEwOuOApSRwYD2YcO0GB0vzz8mR8rxYMjB/W+LDtip4/ZVvpB6cjSh fPMRjIEW rnp/2SZEvPUTqW6tjbt6TeonRBM2fL4ilzNIefHJcOfsXMou2vOUucPMQ91WLZp2fJEcwR3IcPB32Bn694Jbk9LmHncR3X3M1ZF4kYO+TzOXMg+pzuO9KYAg031NsRQjzy733wXWMl5NlSrO3cjZ3h9HSFg/TnQdQuACEsfn47v4NG95JyXL7yfphqtXgxz7qKfPTRda6LpXfh4ANZTyrFXET22vaGeXFO7Of+rBXqDqUuAjAyqHW1jLFnDjRnzelkJvKBh6P/MYBC9biVNYKEBd1DNxf+QVlw4EhFuvtoxdhcTOEMF+qyLloXrEueTF1rXIbut9oV7IqlGAZBbprnz8e2w7+BBNTho06TVbc+mdcgxOFuCDIHznu8Wq5knUcpdFlPXhw/34Q7kOh8DPkDM0u6k73bcppVdsrcpGZyWyPV0U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The fullness_group enum is nested (sub-enum) within the class_stat_type enum. zsmalloc requires the values in both enums to match, because zsmalloc passes these values to generic functions, e.g. class_stat_inc() and class_stat_dec(), after casting them to integers. Replace these enums (and enum nesting) and use simple defines instead. Also rename some of zsmalloc stats defines, as they sort of clash with zspage object tags. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 104 ++++++++++++++++++++++---------------------------- 1 file changed, 45 insertions(+), 59 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index b57a89ed6f30..38ae8963c0eb 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -159,26 +159,18 @@ #define ZS_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \ ZS_SIZE_CLASS_DELTA) + 1) -enum fullness_group { - ZS_EMPTY, - ZS_ALMOST_EMPTY, - ZS_ALMOST_FULL, - ZS_FULL, - NR_ZS_FULLNESS, -}; +#define ZS_EMPTY 0 +#define ZS_ALMOST_EMPTY 1 +#define ZS_ALMOST_FULL 2 +#define ZS_FULL 3 +#define ZS_OBJS_ALLOCATED 4 +#define ZS_OBJS_INUSE 5 -enum class_stat_type { - CLASS_EMPTY, - CLASS_ALMOST_EMPTY, - CLASS_ALMOST_FULL, - CLASS_FULL, - OBJ_ALLOCATED, - OBJ_USED, - NR_ZS_STAT_TYPE, -}; +#define NR_ZS_STAT 6 +#define NR_ZS_FULLNESS 4 struct zs_size_stat { - unsigned long objs[NR_ZS_STAT_TYPE]; + unsigned long objs[NR_ZS_STAT]; }; #ifdef CONFIG_ZSMALLOC_STAT @@ -547,8 +539,8 @@ static inline void set_freeobj(struct zspage *zspage, unsigned int obj) } static void get_zspage_mapping(struct zspage *zspage, - unsigned int *class_idx, - enum fullness_group *fullness) + unsigned int *class_idx, + int *fullness) { BUG_ON(zspage->magic != ZSPAGE_MAGIC); @@ -557,14 +549,14 @@ static void get_zspage_mapping(struct zspage *zspage, } static struct size_class *zspage_class(struct zs_pool *pool, - struct zspage *zspage) + struct zspage *zspage) { return pool->size_class[zspage->class]; } static void set_zspage_mapping(struct zspage *zspage, - unsigned int class_idx, - enum fullness_group fullness) + unsigned int class_idx, + int fullness) { zspage->class = class_idx; zspage->fullness = fullness; @@ -588,23 +580,20 @@ static int get_size_class_index(int size) return min_t(int, ZS_SIZE_CLASSES - 1, idx); } -/* type can be of enum type class_stat_type or fullness_group */ static inline void class_stat_inc(struct size_class *class, - int type, unsigned long cnt) + int type, unsigned long cnt) { class->stats.objs[type] += cnt; } -/* type can be of enum type class_stat_type or fullness_group */ static inline void class_stat_dec(struct size_class *class, - int type, unsigned long cnt) + int type, unsigned long cnt) { class->stats.objs[type] -= cnt; } -/* type can be of enum type class_stat_type or fullness_group */ static inline unsigned long zs_stat_get(struct size_class *class, - int type) + int type) { return class->stats.objs[type]; } @@ -652,10 +641,10 @@ static int zs_stats_size_show(struct seq_file *s, void *v) continue; spin_lock(&pool->lock); - class_almost_full = zs_stat_get(class, CLASS_ALMOST_FULL); - class_almost_empty = zs_stat_get(class, CLASS_ALMOST_EMPTY); - obj_allocated = zs_stat_get(class, OBJ_ALLOCATED); - obj_used = zs_stat_get(class, OBJ_USED); + class_almost_full = zs_stat_get(class, ZS_ALMOST_FULL); + class_almost_empty = zs_stat_get(class, ZS_ALMOST_EMPTY); + obj_allocated = zs_stat_get(class, ZS_OBJS_ALLOCATED); + obj_used = zs_stat_get(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); spin_unlock(&pool->lock); @@ -731,11 +720,10 @@ static inline void zs_pool_stat_destroy(struct zs_pool *pool) * the pool (not yet implemented). This function returns fullness * status of the given page. */ -static enum fullness_group get_fullness_group(struct size_class *class, - struct zspage *zspage) +static int get_fullness_group(struct size_class *class, struct zspage *zspage) { int inuse, objs_per_zspage; - enum fullness_group fg; + int fg; inuse = get_zspage_inuse(zspage); objs_per_zspage = class->objs_per_zspage; @@ -754,11 +742,11 @@ static enum fullness_group get_fullness_group(struct size_class *class, /* * This function adds the given zspage to the fullness list identified - * by . + * by . */ static void insert_zspage(struct size_class *class, struct zspage *zspage, - enum fullness_group fullness) + int fullness) { class_stat_inc(class, fullness, 1); list_add(&zspage->list, &class->fullness_list[fullness]); @@ -766,11 +754,11 @@ static void insert_zspage(struct size_class *class, /* * This function removes the given zspage from the fullness list identified - * by . + * by . */ static void remove_zspage(struct size_class *class, struct zspage *zspage, - enum fullness_group fullness) + int fullness) { VM_BUG_ON(list_empty(&class->fullness_list[fullness])); @@ -787,11 +775,10 @@ static void remove_zspage(struct size_class *class, * page from the freelist of the old fullness group to that of the new * fullness group. */ -static enum fullness_group fix_fullness_group(struct size_class *class, - struct zspage *zspage) +static int fix_fullness_group(struct size_class *class, struct zspage *zspage) { int class_idx; - enum fullness_group currfg, newfg; + int currfg, newfg; get_zspage_mapping(zspage, &class_idx, &currfg); newfg = get_fullness_group(class, zspage); @@ -964,7 +951,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { struct page *page, *next; - enum fullness_group fg; + int fg; unsigned int class_idx; get_zspage_mapping(zspage, &class_idx, &fg); @@ -990,7 +977,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, cache_free_zspage(pool, zspage); - class_stat_dec(class, OBJ_ALLOCATED, class->objs_per_zspage); + class_stat_dec(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated); } @@ -1508,7 +1495,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) { unsigned long handle, obj; struct size_class *class; - enum fullness_group newfg; + int newfg; struct zspage *zspage; if (unlikely(!size || size > ZS_MAX_ALLOC_SIZE)) @@ -1530,7 +1517,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* Now move the zspage to another fullness group, if required */ fix_fullness_group(class, zspage); record_obj(handle, obj); - class_stat_inc(class, OBJ_USED, 1); + class_stat_inc(class, ZS_OBJS_INUSE, 1); spin_unlock(&pool->lock); return handle; @@ -1552,8 +1539,8 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) record_obj(handle, obj); atomic_long_add(class->pages_per_zspage, &pool->pages_allocated); - class_stat_inc(class, OBJ_ALLOCATED, class->objs_per_zspage); - class_stat_inc(class, OBJ_USED, 1); + class_stat_inc(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); + class_stat_inc(class, ZS_OBJS_INUSE, 1); /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); @@ -1609,7 +1596,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) struct page *f_page; unsigned long obj; struct size_class *class; - enum fullness_group fullness; + int fullness; if (IS_ERR_OR_NULL((void *)handle)) return; @@ -1624,7 +1611,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) zspage = get_zspage(f_page); class = zspage_class(pool, zspage); - class_stat_dec(class, OBJ_USED, 1); + class_stat_dec(class, ZS_OBJS_INUSE, 1); #ifdef CONFIG_ZPOOL if (zspage->under_reclaim) { @@ -1828,7 +1815,7 @@ static struct zspage *isolate_zspage(struct size_class *class, bool source) { int i; struct zspage *zspage; - enum fullness_group fg[2] = {ZS_ALMOST_EMPTY, ZS_ALMOST_FULL}; + int fg[2] = {ZS_ALMOST_EMPTY, ZS_ALMOST_FULL}; if (!source) { fg[0] = ZS_ALMOST_FULL; @@ -1852,12 +1839,11 @@ static struct zspage *isolate_zspage(struct size_class *class, bool source) * @class: destination class * @zspage: target page * - * Return @zspage's fullness_group + * Return @zspage's fullness status */ -static enum fullness_group putback_zspage(struct size_class *class, - struct zspage *zspage) +static int putback_zspage(struct size_class *class, struct zspage *zspage) { - enum fullness_group fullness; + int fullness; fullness = get_fullness_group(class, zspage); insert_zspage(class, zspage, fullness); @@ -2121,7 +2107,7 @@ static void async_free_zspage(struct work_struct *work) int i; struct size_class *class; unsigned int class_idx; - enum fullness_group fullness; + int fullness; struct zspage *zspage, *tmp; LIST_HEAD(free_pages); struct zs_pool *pool = container_of(work, struct zs_pool, @@ -2190,8 +2176,8 @@ static inline void zs_flush_migration(struct zs_pool *pool) { } static unsigned long zs_can_compact(struct size_class *class) { unsigned long obj_wasted; - unsigned long obj_allocated = zs_stat_get(class, OBJ_ALLOCATED); - unsigned long obj_used = zs_stat_get(class, OBJ_USED); + unsigned long obj_allocated = zs_stat_get(class, ZS_OBJS_ALLOCATED); + unsigned long obj_used = zs_stat_get(class, ZS_OBJS_INUSE); if (obj_allocated <= obj_used) return 0; @@ -2616,7 +2602,7 @@ static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries) unsigned long handle; struct zspage *zspage; struct page *page; - enum fullness_group fullness; + int fullness; /* Lock LRU and fullness list */ spin_lock(&pool->lock); From patchwork Thu Feb 23 03:04:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13149803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E595C636D6 for ; Thu, 23 Feb 2023 03:05:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB4616B0082; Wed, 22 Feb 2023 22:05:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E64B16B0083; Wed, 22 Feb 2023 22:05:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D05466B0085; Wed, 22 Feb 2023 22:05:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BEDCD6B0082 for ; Wed, 22 Feb 2023 22:05:09 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9FC03A0F17 for ; Thu, 23 Feb 2023 03:05:09 +0000 (UTC) X-FDA: 80497065138.21.1E7FB08 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf04.hostedemail.com (Postfix) with ESMTP id D2C6F4000A for ; Thu, 23 Feb 2023 03:05:07 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RqUAZ7Nv; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677121507; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8+oVIrjvupRwrVwXlg7PumbsuiNQvLcBtQYCdoukFrI=; b=rJQlCCFBhONLjLBepvKfyFO3JB7DTFOXWqYPdhxbVEQnao9KJI1KvxXWdBDX2LSQk1MUO3 ijpvI+CI900uzFlruVDiimwn7x2KMlQz0YFtZ8g0c8RSMSQpAKx/AkD2UyedQw8gZXRbKg PiNfHIWdMWkaajoIlQnIZeyNqcL7Ydg= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RqUAZ7Nv; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677121507; a=rsa-sha256; cv=none; b=CPhMY1jbmujnIzt6R1amqOwGvZkoxMnPKLNSdwajUVLPlGZGzHDbAAijsK/G3ubqd1GZjZ 3kZPd0/8pA2vGbvUc3JKHDdVaBtI/qi2tna0Gu3sS9kVoK+dgG0L1LePSL3dG4LUsxkKUP edDlS4rtTkWtnKs7GKQEGPCRVLsWWwc= Received: by mail-pj1-f43.google.com with SMTP id qa18-20020a17090b4fd200b0023750b675f5so3027608pjb.3 for ; Wed, 22 Feb 2023 19:05:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8+oVIrjvupRwrVwXlg7PumbsuiNQvLcBtQYCdoukFrI=; b=RqUAZ7NvMZZunMUB/MmiD0jYODuJNYZpUHSO0ZTUtA0mNcmpTGprRdyBcM13rR+YkV BqEL4V2jbOwYKfcCfOjCE6vYdJMPvmebObR8Rn82FOYHiwGm/nz48QVNQY69a7MM+Xgi jsEu9m1WWpq9GBL2stEA0/m3bW+gDboZWkQZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8+oVIrjvupRwrVwXlg7PumbsuiNQvLcBtQYCdoukFrI=; b=rLx8d3o/CzFRb7BBm0dWf1ziVS4DkEFkhYsIDuEydfaDKhXJAxARXFy53V7RiQupoI 3JObl7SeCkuhNWW1G1ZIKSXt5CeFX7nK3Rtw19vc7y+R/jbnf2MyxefvtXGUioz7JTxt TPzjU8QlDJ2RkY+isVBQU5yv1nDPKfS2bCe6oWLdWBf45rxWEisWETHL7+uFBKb3qEMY 9DSxEhQASGlN10otObajCueP/MaruKaJ/PICIuwxtCuca912dqZgCiTLeNQAeNPlnb4a bnChcV4GemvHpl9rNk+XMTdPRNeHvgmHMo3RDDNpOG4yowFobtqD2ll2ema5/tkoum9w mOzg== X-Gm-Message-State: AO0yUKUhGqTKupAVdQOpvzt8YlHAzXs3l1HPExBzkeq1BBRLr8MNYnk7 ydwCWCmMoRqV7nlY7t2z7t/8Cw== X-Google-Smtp-Source: AK7set9n/kR4OsBAyMS4NO+Nfr/q6P17NvG3FPUw+0WktVTf7LO65SnlQ+Sk3o2XFtmQGImRRC8Scg== X-Received: by 2002:a17:902:ea07:b0:19a:887d:98ac with SMTP id s7-20020a170902ea0700b0019a887d98acmr13433066plg.46.1677121506725; Wed, 22 Feb 2023 19:05:06 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6de2:9e85:b508:57b8]) by smtp.gmail.com with ESMTPSA id jl21-20020a170903135500b0019926c77577sm608520plb.90.2023.02.22.19.05.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 19:05:06 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv2 3/6] zsmalloc: fine-grained inuse ratio based fullness grouping Date: Thu, 23 Feb 2023 12:04:48 +0900 Message-Id: <20230223030451.543162-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog In-Reply-To: <20230223030451.543162-1-senozhatsky@chromium.org> References: <20230223030451.543162-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D2C6F4000A X-Stat-Signature: ptodepmry8o177obikze5cptxirgf6ia X-Rspam-User: X-HE-Tag: 1677121507-154801 X-HE-Meta: U2FsdGVkX1/t7vpRk/XBUA4leKWTCKSzD8xaofb1DoN2yCG6plnjZ0l58mA0sjTorJuOuiWTyCaEdu8SduQeGmktZZjSKRhw6rFQUIdwyDeSCZn/ix3Pv9uzwtafZjwMYeCoEw4yZLlfzp0NJ/M5K432xQeFg8C1bP0C8athlYKpR2FrLOLaXgY9RJlkICftdMujDVHjcf56tEsy5wpDaNAkaF+EXoCkG26G1r93LTLrQJqBi0daTtW94oiMmGt3JhHM43qP39NY/hVCpv7IEDdBE2TzXm0w9UeJ1eksFp1bd24g5UuiESTQk+bbrnXLCOC04vbfQ6PFwq9RJx4U00peraxt3uzOoEhLjcUFDum3GKstYrK1IusaywrU0ivMQCXxGmELf72/En9rg3jW1W5A2PfmZI2XniZAhAXY9YGsM7IaxEEOD1d6EMn+8FJgZzayYvCfsvbwRBZCfq/Mc+yVg36QHWHc9gSYBN8aZfP2isWsozOV8z9iactcqvfY/yiDavohu13JoEdnAh9OHGZut3x+7bayBRh2IVTU0AYDT63+YXk1tGtkjPWtQDRFxYkxzMgCihNMv9rT0aSbS2UL9/W28N6Rn3+LNNjdHMRu5OigO5bfsH+Uaji2WP58+IoPgMSOWRb+EJMR/qNVFrlBTtGC0W9mRxP9lpByum3jWIEUUdq26ncLrhIyGovxkE7AoW7fWuhh2N7p1fI4IGww+LknJqjjMWjjEzxi+PutVireLf/k3vJB+1RUpbrPVgTYEFnJQ4d7vI9hsIxYmBb/osDLwCxPTg2dSbK6yLw6gxfJNMBv68RbZ0k6pID0TCRCTkAI1cGpdMpVlfdIGmV4yzwMk7jRMWXucGlmNVrtAh6u1ZG3Z68+eMkJ41O+9AEnicEHVqekcs6lDUpEZdp6az0AZ3deDsJtf0iYIEErRDssYp7YL9GZ4cx9Z1ZUVu/q3L7CCzT4Pmgf+yB kEMRyayr vE5NQuoCHM1bZ7iH3zUEBtX6+OqcfguXucikiLDVt5FYNCztlA8cNE0Wy11Tce+Z2pZwmO9hbwgnvSjdnysqvvk7b8JK3bZDn8vPL07SaAVEEAhXpEg6LQtrEyY1ea32L2h2+zgtyzhB6uUMPMsdGWcdfdHKnSNcMra4bqYySkm9La790qzwbN1TLP/z3FfA4SxOfQTbLmH89rFGncff7qD/CAWThra6DMBdxWRdMv3fgLx+9LxnykFtvWa2dcTifKX1CcouguqB5Wt+1m7e94nTV8gCRRr6rqAWJx1sVaOkldwAH3JmEp7qjEAxtJ0q3KB1v8U4FmLRToQ+dNG8Az4WBZl8WGL3uK3JNKeLlBbINrB1rKFdU16tE1zpBPvZxpBmvc1gtrOm2rtmT+Ig73zLlbnk/68CswO56C/UTNGvLZ/NdCSWY5mCOd6aOcqqsObx5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Each zspage maintains ->inuse counter which keeps track of the number of objects stored in the page. The ->inuse counter also determines the page's "fullness group" which is calculated as the ratio of the "inuse" objects to the total number of objects the page can hold (objs_per_zspage). The closer the ->inuse counter is to objs_per_zspage, the better. Each size class maintains several fullness lists, that keep track of zspages of particular "fullness". Pages within each fullness list are stored in random order with regard to the ->inuse counter. This is because sorting the pages by ->inuse counter each time obj_malloc() or obj_free() is called would be too expensive. However, the ->inuse counter is still a crucial factor in many situations. For the two major zsmalloc operations, zs_malloc() and zs_compact(), we typically select the head page from the corresponding fullness list as the best candidate page. However, this assumption is not always accurate. For the zs_malloc() operation, the optimal candidate page should have the highest ->inuse counter. This is because the goal is to maximize the number of ZS_FULL pages and make full use of all allocated memory. For the zs_compact() operation, the optimal candidate page should have the lowest ->inuse counter. This is because compaction needs to move objects in use to another page before it can release the zspage and return its physical pages to the buddy allocator. The fewer objects in use, the quicker compaction can release the page. Additionally, compaction is measured by the number of pages it releases. This patch reworks the fullness grouping mechanism. Instead of having two groups - ZS_ALMOST_EMPTY (usage ratio below 3/4) and ZS_ALMOST_FULL (usage ration above 3/4) - that result in too many pages being included in the ALMOST_EMPTY group for specific classes, size classes maintain a larger number of fullness lists that give strict guarantees on the minimum and maximum ->inuse values within each group. Each group represents a 10% change in the ->inuse ratio compared to neighboring groups. In essence, there are groups for pages with 0%, 10%, 20% usage ratios, and so on, up to 100%. This enhances the selection of candidate pages for both zs_malloc() and zs_compact(). A printout of the ->inuse counters of the first 7 pages per (random) class fullness group: class-768 objs_per_zspage 16: fullness 100%: empty fullness 99%: empty fullness 90%: empty fullness 80%: empty fullness 70%: empty fullness 60%: 8 8 9 9 8 8 8 fullness 50%: empty fullness 40%: 5 5 6 5 5 5 5 fullness 30%: 4 4 4 4 4 4 4 fullness 20%: 2 3 2 3 3 2 2 fullness 10%: 1 1 1 1 1 1 1 fullness 0%: empty The zs_malloc() function searches through the groups of pages starting with the one having the highest usage ratio. This means that it always selects a page from the group with the least internal fragmentation (highest usage ratio) and makes it even less fragmented by increasing its usage ratio. The zs_compact() function, on the other hand, begins by scanning the group with the highest fragmentation (lowest usage ratio) to locate the source page. The first available page is selected, and then the function moves downward to find a destination page in the group with the lowest internal fragmentation (highest usage ratio). Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 183 +++++++++++++++++++++++++++++--------------------- 1 file changed, 107 insertions(+), 76 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 38ae8963c0eb..1a92ebe338eb 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -127,7 +127,7 @@ #define OBJ_INDEX_MASK ((_AC(1, UL) << OBJ_INDEX_BITS) - 1) #define HUGE_BITS 1 -#define FULLNESS_BITS 2 +#define FULLNESS_BITS 4 #define CLASS_BITS 8 #define ISOLATED_BITS 5 #define MAGIC_VAL_BITS 8 @@ -159,15 +159,33 @@ #define ZS_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \ ZS_SIZE_CLASS_DELTA) + 1) -#define ZS_EMPTY 0 -#define ZS_ALMOST_EMPTY 1 -#define ZS_ALMOST_FULL 2 -#define ZS_FULL 3 -#define ZS_OBJS_ALLOCATED 4 -#define ZS_OBJS_INUSE 5 - -#define NR_ZS_STAT 6 -#define NR_ZS_FULLNESS 4 +/* + * Pages are distinguished by the ratio of used memory (that is the ratio + * of ->inuse objects to all objects that page can store). For example, + * INUSE_RATIO_30 means that the ratio of used objects is > 20% and <= 30%. + * + * The number of fullness groups is not random. It allows us to keep + * diffeence between the least busy page in the group (minimum permitted + * number of ->inuse objects) and the most busy page (maximum permitted + * number of ->inuse objects) at a reasonable value. + */ +#define ZS_INUSE_RATIO_0 0 +#define ZS_INUSE_RATIO_10 1 +#define ZS_INUSE_RATIO_20 2 +#define ZS_INUSE_RATIO_30 3 +#define ZS_INUSE_RATIO_40 4 +#define ZS_INUSE_RATIO_50 5 +#define ZS_INUSE_RATIO_60 6 +#define ZS_INUSE_RATIO_70 7 +#define ZS_INUSE_RATIO_80 8 +#define ZS_INUSE_RATIO_90 9 +#define ZS_INUSE_RATIO_99 10 +#define ZS_INUSE_RATIO_100 11 +#define ZS_OBJS_ALLOCATED 12 +#define ZS_OBJS_INUSE 13 + +#define NR_ZS_INUSE_RATIO 12 +#define NR_ZS_STAT 14 struct zs_size_stat { unsigned long objs[NR_ZS_STAT]; @@ -177,25 +195,10 @@ struct zs_size_stat { static struct dentry *zs_stat_root; #endif -/* - * We assign a page to ZS_ALMOST_EMPTY fullness group when: - * n <= N / f, where - * n = number of allocated objects - * N = total number of objects zspage can store - * f = fullness_threshold_frac - * - * Similarly, we assign zspage to: - * ZS_ALMOST_FULL when n > N / f - * ZS_EMPTY when n == 0 - * ZS_FULL when n == N - * - * (see: fix_fullness_group()) - */ -static const int fullness_threshold_frac = 4; static size_t huge_class_size; struct size_class { - struct list_head fullness_list[NR_ZS_FULLNESS]; + struct list_head fullness_list[NR_ZS_INUSE_RATIO]; /* * Size of objects stored in this class. Must be multiple * of ZS_ALIGN. @@ -641,8 +644,23 @@ static int zs_stats_size_show(struct seq_file *s, void *v) continue; spin_lock(&pool->lock); - class_almost_full = zs_stat_get(class, ZS_ALMOST_FULL); - class_almost_empty = zs_stat_get(class, ZS_ALMOST_EMPTY); + + /* + * Replecate old behaviour for almost_full and almost_empty + * stats. + */ + class_almost_full = zs_stat_get(class, ZS_INUSE_RATIO_99); + class_almost_full += zs_stat_get(class, ZS_INUSE_RATIO_90); + class_almost_full += zs_stat_get(class, ZS_INUSE_RATIO_80); + class_almost_full += zs_stat_get(class, ZS_INUSE_RATIO_70); + + class_almost_empty = zs_stat_get(class, ZS_INUSE_RATIO_60); + class_almost_empty += zs_stat_get(class, ZS_INUSE_RATIO_50); + class_almost_empty += zs_stat_get(class, ZS_INUSE_RATIO_40); + class_almost_empty += zs_stat_get(class, ZS_INUSE_RATIO_30); + class_almost_empty += zs_stat_get(class, ZS_INUSE_RATIO_20); + class_almost_empty += zs_stat_get(class, ZS_INUSE_RATIO_10); + obj_allocated = zs_stat_get(class, ZS_OBJS_ALLOCATED); obj_used = zs_stat_get(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); @@ -712,32 +730,30 @@ static inline void zs_pool_stat_destroy(struct zs_pool *pool) } #endif - /* * For each size class, zspages are divided into different groups - * depending on how "full" they are. This was done so that we could - * easily find empty or nearly empty zspages when we try to shrink - * the pool (not yet implemented). This function returns fullness + * depending on their usage ratio. This function returns fullness * status of the given page. */ static int get_fullness_group(struct size_class *class, struct zspage *zspage) { - int inuse, objs_per_zspage; - int fg; + int inuse, objs_per_zspage, ratio; inuse = get_zspage_inuse(zspage); objs_per_zspage = class->objs_per_zspage; if (inuse == 0) - fg = ZS_EMPTY; - else if (inuse == objs_per_zspage) - fg = ZS_FULL; - else if (inuse <= 3 * objs_per_zspage / fullness_threshold_frac) - fg = ZS_ALMOST_EMPTY; - else - fg = ZS_ALMOST_FULL; + return ZS_INUSE_RATIO_0; + if (inuse == objs_per_zspage) + return ZS_INUSE_RATIO_100; - return fg; + ratio = 100 * inuse / objs_per_zspage; + /* + * Take integer division into consideration: a page with one inuse + * object out of 127 possible, will endup having 0 usage ratio, + * which is wrong as it belongs in ZS_INUSE_RATIO_10 fullness group. + */ + return ratio / 10 + 1; } /* @@ -769,11 +785,11 @@ static void remove_zspage(struct size_class *class, /* * Each size class maintains zspages in different fullness groups depending * on the number of live objects they contain. When allocating or freeing - * objects, the fullness status of the page can change, say, from ALMOST_FULL - * to ALMOST_EMPTY when freeing an object. This function checks if such - * a status change has occurred for the given page and accordingly moves the - * page from the freelist of the old fullness group to that of the new - * fullness group. + * objects, the fullness status of the page can change, for instance, from + * INUSE_RATIO_80 to INUSE_RATIO_70 when freeing an object. This function + * checks if such a status change has occurred for the given page and + * accordingly moves the page from the list of the old fullness group to that + * of the new fullness group. */ static int fix_fullness_group(struct size_class *class, struct zspage *zspage) { @@ -959,7 +975,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, assert_spin_locked(&pool->lock); VM_BUG_ON(get_zspage_inuse(zspage)); - VM_BUG_ON(fg != ZS_EMPTY); + VM_BUG_ON(fg != ZS_INUSE_RATIO_0); /* Free all deferred handles from zs_free */ free_handles(pool, class, zspage); @@ -998,7 +1014,7 @@ static void free_zspage(struct zs_pool *pool, struct size_class *class, return; } - remove_zspage(class, zspage, ZS_EMPTY); + remove_zspage(class, zspage, ZS_INUSE_RATIO_0); #ifdef CONFIG_ZPOOL list_del(&zspage->lru); #endif @@ -1134,9 +1150,9 @@ static struct zspage *find_get_zspage(struct size_class *class) int i; struct zspage *zspage; - for (i = ZS_ALMOST_FULL; i >= ZS_EMPTY; i--) { + for (i = ZS_INUSE_RATIO_99; i >= ZS_INUSE_RATIO_0; i--) { zspage = list_first_entry_or_null(&class->fullness_list[i], - struct zspage, list); + struct zspage, list); if (zspage) break; } @@ -1629,7 +1645,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_free(class->size, obj, NULL); fullness = fix_fullness_group(class, zspage); - if (fullness == ZS_EMPTY) + if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); spin_unlock(&pool->lock); @@ -1811,22 +1827,33 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, return ret; } -static struct zspage *isolate_zspage(struct size_class *class, bool source) +static struct zspage *isolate_src_zspage(struct size_class *class) { - int i; struct zspage *zspage; - int fg[2] = {ZS_ALMOST_EMPTY, ZS_ALMOST_FULL}; + int fg; - if (!source) { - fg[0] = ZS_ALMOST_FULL; - fg[1] = ZS_ALMOST_EMPTY; + for (fg = ZS_INUSE_RATIO_10; fg <= ZS_INUSE_RATIO_99; fg++) { + zspage = list_first_entry_or_null(&class->fullness_list[fg], + struct zspage, list); + if (zspage) { + remove_zspage(class, zspage, fg); + return zspage; + } } - for (i = 0; i < 2; i++) { - zspage = list_first_entry_or_null(&class->fullness_list[fg[i]], - struct zspage, list); + return zspage; +} + +static struct zspage *isolate_dst_zspage(struct size_class *class) +{ + struct zspage *zspage; + int fg; + + for (fg = ZS_INUSE_RATIO_99; fg >= ZS_INUSE_RATIO_10; fg--) { + zspage = list_first_entry_or_null(&class->fullness_list[fg], + struct zspage, list); if (zspage) { - remove_zspage(class, zspage, fg[i]); + remove_zspage(class, zspage, fg); return zspage; } } @@ -2119,7 +2146,7 @@ static void async_free_zspage(struct work_struct *work) continue; spin_lock(&pool->lock); - list_splice_init(&class->fullness_list[ZS_EMPTY], &free_pages); + list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); spin_unlock(&pool->lock); } @@ -2128,7 +2155,7 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); get_zspage_mapping(zspage, &class_idx, &fullness); - VM_BUG_ON(fullness != ZS_EMPTY); + VM_BUG_ON(fullness != ZS_INUSE_RATIO_0); class = pool->size_class[class_idx]; spin_lock(&pool->lock); #ifdef CONFIG_ZPOOL @@ -2201,7 +2228,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ spin_lock(&pool->lock); - while ((src_zspage = isolate_zspage(class, true))) { + while ((src_zspage = isolate_src_zspage(class))) { /* protect someone accessing the zspage(i.e., zs_map_object) */ migrate_write_lock(src_zspage); @@ -2211,7 +2238,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, cc.obj_idx = 0; cc.s_page = get_first_page(src_zspage); - while ((dst_zspage = isolate_zspage(class, false))) { + while ((dst_zspage = isolate_dst_zspage(class))) { migrate_write_lock_nested(dst_zspage); cc.d_page = get_first_page(dst_zspage); @@ -2236,7 +2263,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); migrate_write_unlock(dst_zspage); - if (putback_zspage(class, src_zspage) == ZS_EMPTY) { + if (putback_zspage(class, src_zspage) == ZS_INUSE_RATIO_0) { migrate_write_unlock(src_zspage); free_zspage(pool, class, src_zspage); pages_freed += class->pages_per_zspage; @@ -2394,7 +2421,7 @@ struct zs_pool *zs_create_pool(const char *name) int pages_per_zspage; int objs_per_zspage; struct size_class *class; - int fullness = 0; + int fullness; size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA; if (size > ZS_MAX_ALLOC_SIZE) @@ -2448,9 +2475,12 @@ struct zs_pool *zs_create_pool(const char *name) class->pages_per_zspage = pages_per_zspage; class->objs_per_zspage = objs_per_zspage; pool->size_class[i] = class; - for (fullness = ZS_EMPTY; fullness < NR_ZS_FULLNESS; - fullness++) + + fullness = ZS_INUSE_RATIO_0; + while (fullness < NR_ZS_INUSE_RATIO) { INIT_LIST_HEAD(&class->fullness_list[fullness]); + fullness++; + } prev_class = class; } @@ -2496,11 +2526,12 @@ void zs_destroy_pool(struct zs_pool *pool) if (class->index != i) continue; - for (fg = ZS_EMPTY; fg < NR_ZS_FULLNESS; fg++) { - if (!list_empty(&class->fullness_list[fg])) { - pr_info("Freeing non-empty class with size %db, fullness group %d\n", - class->size, fg); - } + for (fg = ZS_INUSE_RATIO_0; fg < NR_ZS_INUSE_RATIO; fg++) { + if (list_empty(&class->fullness_list[fg])) + continue; + + pr_err("Class-%d fullness group %d is not empty\n", + class->size, fg); } kfree(class); } @@ -2672,7 +2703,7 @@ static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries) * while the page is removed from the pool. Fix it * up for the check in __free_zspage(). */ - zspage->fullness = ZS_EMPTY; + zspage->fullness = ZS_INUSE_RATIO_0; __free_zspage(pool, class, zspage); spin_unlock(&pool->lock); From patchwork Thu Feb 23 03:04:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13149804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 668BEC64ED6 for ; Thu, 23 Feb 2023 03:05:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00F3B6B0083; Wed, 22 Feb 2023 22:05:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F00666B0085; Wed, 22 Feb 2023 22:05:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D535E6B0087; Wed, 22 Feb 2023 22:05:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C4FCB6B0083 for ; Wed, 22 Feb 2023 22:05:12 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8D550A0F15 for ; Thu, 23 Feb 2023 03:05:12 +0000 (UTC) X-FDA: 80497065264.30.729685F Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf28.hostedemail.com (Postfix) with ESMTP id A3793C0010 for ; Thu, 23 Feb 2023 03:05:10 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=FukfwasN; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677121510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GiCIpL03JBPiX3r0aUEMNZiww2WahzTmtx5CHsJtG58=; b=4pZY+FA1cyA5B4rR7BWwG5WotD+LrlIFpASwP9EL8ZqnoXu0qOqAJnvEoqY5DduKF8pihq O9/qCDOZCFGWbLzTMbqRGNl75LHU300rW17SWtJLOYhcIXgRhcDEsSBHyU1SqtXyHZmheP YAte3NZ6+R+vhWLcfm4xbL4XvmxJ/aA= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=FukfwasN; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677121510; a=rsa-sha256; cv=none; b=C5WP1iBSpCRT+pDWyDH2VU6xyFSudIKLY9UZ7+f3ZU2sjAhYBg/3U2TunH653MXc9PVugl leOlwE+tAHAmzNnjk8E0MZU3/had7F6AUA70ruvEgpkWm9f++GxbPFuYOotPuRhaUERjUO 0+MB5Yc0B1Lz61DqJ4sOxUiTsgWl4iQ= Received: by mail-pl1-f182.google.com with SMTP id ky4so12450731plb.3 for ; Wed, 22 Feb 2023 19:05:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GiCIpL03JBPiX3r0aUEMNZiww2WahzTmtx5CHsJtG58=; b=FukfwasNWu2Rzz92i5So8JtB90Utw+Izt47hiVM88qDg1BDIfH+2JocdZt1RMXn+V/ 4dm6ceSVu9mc/bdHNTCeWGCNCPyvnKlp70VGAgY9rIQj+GssKAweQyHmVuNintyLSZId fBGzg7ROy/QxWOcEJqobjmpM/xp/ZJgkisuco= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GiCIpL03JBPiX3r0aUEMNZiww2WahzTmtx5CHsJtG58=; b=TYYe1GGYlvwe35btmUoBsWQcIuMCQUjNZfmrOkdEv+MDachh7vCLSK20WIx+yYXUIB D/8SdxdqY3RzsBXgMfu3IUDQUSyGThy6ibl4i2L791H3gl2qqj6qBeVN0l/SvXr5licB mVCZU/wrYimp8jlNRNokSiw9EX5+WDcxXUFLIcvLytPqePcJyAeoFHhVO/RPYaeQ/QJR w5wIXdulVbo2tqybN3cHOCSCKUMCsDydmzudfC7jbSKwtbg9VGy572BLq8ZLNG8+gQs+ w2JtU+tn44LEUcopqqZw56l4OyEOg4J8bRIhv4dICy6qMCms4SXfyxCR26cwiuw2tl/T s0iA== X-Gm-Message-State: AO0yUKV+mpqtyURDWPCYeQ0JdiRt8uMrcswsoEEZ/TiBL4EGaeR+JPmT AnShhH4xvOOVUNLWEHtfZ7sTpw== X-Google-Smtp-Source: AK7set8sb4kTJVjexH2DXpQTOdukrnurX2LTxpUBToPlcF9kX7HONNbD7HTaLTBbcSMjfIIg4eb4VA== X-Received: by 2002:a17:903:284:b0:19a:9797:1631 with SMTP id j4-20020a170903028400b0019a97971631mr10850314plr.3.1677121509605; Wed, 22 Feb 2023 19:05:09 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6de2:9e85:b508:57b8]) by smtp.gmail.com with ESMTPSA id jl21-20020a170903135500b0019926c77577sm608520plb.90.2023.02.22.19.05.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 19:05:09 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv2 4/6] zsmalloc: rework compaction algorithm Date: Thu, 23 Feb 2023 12:04:49 +0900 Message-Id: <20230223030451.543162-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog In-Reply-To: <20230223030451.543162-1-senozhatsky@chromium.org> References: <20230223030451.543162-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A3793C0010 X-Rspam-User: X-Stat-Signature: q889duh9ieyyrkb4krm94rq9frwxypro X-HE-Tag: 1677121510-956662 X-HE-Meta: U2FsdGVkX19h0ZhY5+ZEXrOTDJZnZyw/9FNm/cFF7hdY3JQ5bW+078z7E5C1i1LSFmkziy0UvHfik0NKdpEX2Aaips8fN7fUKgNfvQbVmGrIkAbL/mIC+0k7bNSoz2OPyCXmHiGGc3AIFpuzBTNHLFKdhEkLs9u6dmCFyV17iYZ//0el21L/Lymbvh7P1iwg4/QdF7KuZiqPyRzM2jmW6Mm/5tX2d/X0ZLDszaajeLyO2K1yfXuTe/NrdaYuRnx92iVabVYGHZF6j/IsNp4zpO/cKps+BDo9lowElpkyij4jPqeIbMe1c5UfJMTrtqA4NWulVtLSoKwEfqAG+kRk2BJxZhA1fAZ82xdkJQBOzLFSrjGL8D9cJHjRqjNDt3O1XFgYvdXq8MBjTQzMTB1nyYSARWPxw3jJ3fHWM/vbljfNMKAt2OE3OqhKdQTtel237vyttuoCnAZhXcI3tYf5XrS5fkLfZp5xm9nrLKdss3zFAgN/J47S/BWrQV7f1Yh1bjbxz6ArVHjbv0RW4kt/L9jkIQQINWbFvTvD41go1Z86CNWoFtJF7RWfZhnYoibO9F1bR9GH3JAaOkx2V69xn6vB1EUesWGqAxr3JE0yCZ2dt2phvBbCi3DLEvOikke3uR7j0uPK28+WDdvuVNg3ZIMEz4Cwl3PNL7jwlP/lyc+eHEAsz59EvX7jZKzZvPprx0Jv9PZHLw1d9hU8JICjHWBm22oFjzSdyCDdcJiNPGjQqIMqvRo0fRMo5Lk8/GoV/K+QlA3lTeRwFR6RHqprwioXDg44jSJh4JL316BWbHGoogQTE9E1CXleo/X0G5ImHVHP+k4kCtrEXeijtGKnkPeRDMZZ/cndixB/PVPj2vcRHOnymR6/T0jJ6h2H8C2TsLVJwS+dAbUqTbiR5FY2nubT7R8hX4AnQE8cNpq7d2nCj4j38ecHds/PfM+AJ5RuTJqPp764SKarD3SN5I4 OT/3wd4D zXruhXgmktJWSmRb33FuVs9iAQE0iL2PqY2cmlDMMmb4+wTU5w1fMP9zkZ4EoRol0CkPzMI8sy6o483Iq5mdEx3MW2Wqx6kMjeTBRg9cWzwXRzxpvLl/OB9rVOmwOxdZDGJFmqDdN6rtRdCxq4lj0Q+Gnyp0zAc/wEtVbct/VLRV2X63rd/wTSZucWqsQFD1bG+YiudZJNDUBFN67oVt5+xLq2HmSiwzmRC82qq1SMkNwKAhj9WxTEzVOwW2+TK2ujdwJlhF4/Pn6PAWRPYaHSMR7UDsMvVl37yglMm8ujbOw1NdnftAKoM3IL02U8BBS6HINqWHOPgs1YUtSKQUhlz5C3r6T4DBG9hKn64ulogSM7nOg3tG64DvH9NeufAIzcKP7QdVjY10R+1U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.003454, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The zsmalloc compaction algorithm has the potential to waste some CPU cycles, particularly when compacting pages within the same fullness group. This is due to the way it selects the head page of the fullness list for source and destination pages, and how it reinserts those pages during each iteration. The algorithm may first use a page as a migration destination and then as a migration source, leading to an unnecessary back-and-forth movement of objects. Consider the following fullness list: PageA PageB PageC PageD PageE During the first iteration, the compaction algorithm will select PageA as the source and PageB as the destination. All of PageA's objects will be moved to PageB, and then PageA will be released while PageB is reinserted into the fullness list. PageB PageC PageD PageE During the next iteration, the compaction algorithm will again select the head of the list as the source and destination, meaning that PageB will now serve as the source and PageC as the destination. This will result in the objects being moved away from PageB, the same objects that were just moved to PageB in the previous iteration. To prevent this avalanche effect, the compaction algorithm should not reinsert the destination page between iterations. By doing so, the most optimal page will continue to be used and its usage ratio will increase, reducing internal fragmentation. The destination page should only be reinserted into the fullness list if: - It becomes full - No source page is available. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 82 ++++++++++++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 44 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 1a92ebe338eb..eacf9e32da5c 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1786,15 +1786,14 @@ struct zs_compact_control { int obj_idx; }; -static int migrate_zspage(struct zs_pool *pool, struct size_class *class, - struct zs_compact_control *cc) +static void migrate_zspage(struct zs_pool *pool, struct size_class *class, + struct zs_compact_control *cc) { unsigned long used_obj, free_obj; unsigned long handle; struct page *s_page = cc->s_page; struct page *d_page = cc->d_page; int obj_idx = cc->obj_idx; - int ret = 0; while (1) { handle = find_alloced_obj(class, s_page, &obj_idx); @@ -1807,10 +1806,8 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, } /* Stop if there is no more space */ - if (zspage_full(class, get_zspage(d_page))) { - ret = -ENOMEM; + if (zspage_full(class, get_zspage(d_page))) break; - } used_obj = handle_to_obj(handle); free_obj = obj_malloc(pool, get_zspage(d_page), handle); @@ -1823,8 +1820,6 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, /* Remember last position in this iteration */ cc->s_page = s_page; cc->obj_idx = obj_idx; - - return ret; } static struct zspage *isolate_src_zspage(struct size_class *class) @@ -2228,57 +2223,56 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ spin_lock(&pool->lock); - while ((src_zspage = isolate_src_zspage(class))) { - /* protect someone accessing the zspage(i.e., zs_map_object) */ - migrate_write_lock(src_zspage); - - if (!zs_can_compact(class)) - break; - - cc.obj_idx = 0; - cc.s_page = get_first_page(src_zspage); - - while ((dst_zspage = isolate_dst_zspage(class))) { - migrate_write_lock_nested(dst_zspage); - + while (1) { + if (!dst_zspage) { + dst_zspage = isolate_dst_zspage(class); + if (!dst_zspage) + goto out; + migrate_write_lock(dst_zspage); cc.d_page = get_first_page(dst_zspage); - /* - * If there is no more space in dst_page, resched - * and see if anyone had allocated another zspage. - */ - if (!migrate_zspage(pool, class, &cc)) - break; + } + if (!zs_can_compact(class)) { putback_zspage(class, dst_zspage); migrate_write_unlock(dst_zspage); - dst_zspage = NULL; - if (spin_is_contended(&pool->lock)) - break; + goto out; } - /* Stop if we couldn't find slot */ - if (dst_zspage == NULL) - break; + src_zspage = isolate_src_zspage(class); + if (!src_zspage) { + putback_zspage(class, dst_zspage); + migrate_write_unlock(dst_zspage); + goto out; + } - putback_zspage(class, dst_zspage); - migrate_write_unlock(dst_zspage); + migrate_write_lock_nested(src_zspage); + + cc.obj_idx = 0; + cc.s_page = get_first_page(src_zspage); + migrate_zspage(pool, class, &cc); if (putback_zspage(class, src_zspage) == ZS_INUSE_RATIO_0) { migrate_write_unlock(src_zspage); free_zspage(pool, class, src_zspage); pages_freed += class->pages_per_zspage; - } else + } else { migrate_write_unlock(src_zspage); - spin_unlock(&pool->lock); - cond_resched(); - spin_lock(&pool->lock); - } + } - if (src_zspage) { - putback_zspage(class, src_zspage); - migrate_write_unlock(src_zspage); - } + if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 + || spin_is_contended(&pool->lock)) { + putback_zspage(class, dst_zspage); + migrate_write_unlock(dst_zspage); + dst_zspage = NULL; + } + if (!dst_zspage) { + spin_unlock(&pool->lock); + cond_resched(); + spin_lock(&pool->lock); + } + } +out: spin_unlock(&pool->lock); return pages_freed; From patchwork Thu Feb 23 03:04:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13149805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4F5EC61DA4 for ; Thu, 23 Feb 2023 03:05:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 628F06B0085; Wed, 22 Feb 2023 22:05:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B2EA6B0087; Wed, 22 Feb 2023 22:05:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 452BB6B0088; Wed, 22 Feb 2023 22:05:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 38C3F6B0085 for ; Wed, 22 Feb 2023 22:05:15 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0E6BB16096A for ; Thu, 23 Feb 2023 03:05:15 +0000 (UTC) X-FDA: 80497065390.12.362536D Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf16.hostedemail.com (Postfix) with ESMTP id 400FF180016 for ; Thu, 23 Feb 2023 03:05:13 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=eXWiDf6C; spf=pass (imf16.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.54 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677121513; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BpGG/okMdj5ni9xegYLzCx8QuNxJjUnFEqNX1SXn5aY=; b=dUs+qSuPMgGM4NCyJmHxQxNq7wWLy6w7m9URBoX63RR+09uqGhrwsr24G9quvflmTb7spQ AI/S4rr24Dk4VdCs9wlTpCI9A3XZKco9rTPHhOZwQliIAmSBl8/zDpF0DuAnw+pYWRyYGq K1aaZslMtjq9YISd3qOTggWu4d0bVDU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=eXWiDf6C; spf=pass (imf16.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.54 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677121513; a=rsa-sha256; cv=none; b=v5gSoCOL7oSmJvwk0sXxAT1I0JK2SDWGnHl950ZNjLbW7PSE/bVnEXfUDmozs8uoz8b9F8 xhjkamUD0MzbpKou9ZDmpzrunQol81rKmf0EnF6H+WELNemqv7vO5qgLerqjJCyLPn9rW+ DPWWtu+sYDroaDy8w3R9GMXVV+z1nZQ= Received: by mail-pj1-f54.google.com with SMTP id m3-20020a17090ade0300b00229eec90a7fso3205387pjv.0 for ; Wed, 22 Feb 2023 19:05:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BpGG/okMdj5ni9xegYLzCx8QuNxJjUnFEqNX1SXn5aY=; b=eXWiDf6CYI70E3qaKKdmQKpYXcXl5mJcHwRXnE9PqLSpLMIBznVHG6rsd6Yik0X/kL 7XBQaZKnWMaK+ZCn7rGTemi+XWQrhgo9ktJrz62HQnL1ZI8wqWskCMnCOUoRKkFs/oq2 x5q7YV9lJ90uWhY6HMhvkqjtbT42l++wXe/Sg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BpGG/okMdj5ni9xegYLzCx8QuNxJjUnFEqNX1SXn5aY=; b=J5NzldQCprEHTokb5apTuicxCCp/DMw+DFA8SmWaTOJuAnQDow/fXN8Ms20bRylZxb l9uKUiGDHaL9FrzNGb1n7YmxqGnEO/yNQLFnalmlO4OUE7pn+8PbSQTj+ha1Iz4iuI3s ckfRI50JpFdtycI8qzog0yur2UfqwVW4NdUoaLWmA2NfUU45NgEAQqscu2qg82sQDxDe j/AfDHf22+o9AsKBjpin84XSPy2vF7bjNA0wol0UMcX3wyR/CwAPuRLLbm2sgzlRxna3 YHJO97sKPQPSsca+GzRX7w9THc+X8fLFU0IX0dzMWAdkq5FQqyYcxoc/z6mYkUJiVfJF rpqQ== X-Gm-Message-State: AO0yUKUbRoilcB0ulfwTvWIg1IL+0WCS3Kk69fvFcem63owV+oQYCGaa fqzgT/tX3OA/nU9lozM4YfHXQg== X-Google-Smtp-Source: AK7set+4oxGj/Ama0iVUAf/6wEdS2tLFbPW6Kq4JQCSnEY0zMYJIdlBBjbRva1S9Em30N43trG4ihA== X-Received: by 2002:a17:902:e5c9:b0:19c:355c:6eb5 with SMTP id u9-20020a170902e5c900b0019c355c6eb5mr14014208plf.30.1677121512118; Wed, 22 Feb 2023 19:05:12 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6de2:9e85:b508:57b8]) by smtp.gmail.com with ESMTPSA id jl21-20020a170903135500b0019926c77577sm608520plb.90.2023.02.22.19.05.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 19:05:11 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv2 5/6] zsmalloc: extend compaction statistics Date: Thu, 23 Feb 2023 12:04:50 +0900 Message-Id: <20230223030451.543162-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog In-Reply-To: <20230223030451.543162-1-senozhatsky@chromium.org> References: <20230223030451.543162-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: u4poyrcau14exbibqj9uxyijq97yq9qg X-Rspamd-Queue-Id: 400FF180016 X-HE-Tag: 1677121513-319497 X-HE-Meta: U2FsdGVkX19jyB9AswGSdKSzc6pWIgiuxRcH5YIAAmsDQVtM03r/lnlcz/px8aPwNJXgyT0uO74xwF+FpU8lgJ7jhHjOwlJ9gYpsxUPJkQ7RSxxyXIcPW0ygqiTDGx6mjBkGec+MLwEwUXyXlyLeP7iSq6uPWpUvRKYDIlyxI4djy3f6F2o2sxPsGi3u7mPld8mIqVSDZ9JD9485WfsxtTxCY67eYVDHvXx9uEVCSa8D/c8ZeQKiQr48M5lTU6R0DvNF9ZxuB5S60AyPjSHsFAthlxn89epufowlJlNjnumYiTsnRrZM8JIgC4S9kyV4UN2iRAERjaLfem/HrRxNRF02GRMmNnUb3VINg0t6yvb/ckAAOu6jpU8+gBkTMUPGlzdQd2DGrZ71wlXUsJJhMr5wzms1oC0XDo7NRWKfjwPEPIBWBF1zqA2iDDwAByFlMcKkOYGsJ3v4T2xaY0ImRKxDOgBoxvk7qNwQrmEBYRx+IeE99HcNfzsr3GvsHv4byD/dCmosddLY9p+hkxEw/YCNF6mBHkjwlkOJLDRfHgMEbsc8Eo+xvvqGKEPnt9RC2uCCfMS+tF0u7o7eHmdkftcbXgrdXHSS2us5sIlEnOOpftCNDrV0wKTo1FEL4HOPw1CjeINdTNtB94fgYHBKWzlufkHuZ4/Wn66fOU1axS/4Kxtm1eMoKu1D/NQTppmZ+/dGAPr5g/w0Moy8L/OjA37SFv5mNwuRml98QGr0YHeoyU96C+2PEQn2+YMw5PMBpn3M9PVNQff2dPWpCQ87+Bei6aW/XfkTK3TTdGa8FfzbOQ9NisAIjTXXyeRy6y0kX/RtCd6pzHnqYqBah12hB6Vgh4w/QjHDiEeo4yE2jgziZiGa2T0zV9crAW6g0xnG/ApM7bE5GqupoFfoaPQcx4lvJQPygRoiOWrVoULf1a8jTKG47HlJBYzQyfwHGZeJXNY+ObuQDu7TSBi6fYs /AL+Nymv ox+RA9mbIiiFWKwmyDwwg/xUl3K8ytdVrDaGC3xLRUtEje9kwgv9RF9xZ3HkPUkDeZrdiS/DlDuiP31Ce5TyN8t9+tv/XPWDSOesJP3wm7jUeJVoGh2wDp9tqbRfpFvbsn6/5OkeDPGRXmJBCj3+IMQ8N0oWctRpjvoNgerYPgXaOKBnj3nowIH5SvZvMZy0tmDsPhTESOQCW6GEOAxvZervp4oAHNkZuQSoib1B1LwaH4XOrWyuLfcJ5DIDpTS1aiEtp/8fo/oN/ChGmMrj9m4ig8u0YR1lHs0/15t6YgC1g1f/Jv8E3R9GRu2dyr3xf/G+WKVwIyyhiJvc+puxYA76iymmyERXXb93f+CIdBQDXLeK3NhFnLNiVTm/yjpN9mfa3/KRefnusN8U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Extend zsmalloc zs_pool_stats with a new member that holds the number of objects pool compaction moved between pool pages. Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 2 ++ mm/zsmalloc.c | 1 + 2 files changed, 3 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..8b3fa5b4a68c 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -36,6 +36,8 @@ enum zs_mapmode { struct zs_pool_stats { /* How many pages were migrated (freed) */ atomic_long_t pages_compacted; + /* How many objects were migrated during compaction */ + atomic_long_t objs_moved; }; struct zs_pool; diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index eacf9e32da5c..f7e69df48fb0 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1815,6 +1815,7 @@ static void migrate_zspage(struct zs_pool *pool, struct size_class *class, obj_idx++; record_obj(handle, free_obj); obj_free(class->size, used_obj, NULL); + atomic_long_inc(&pool->stats.objs_moved); } /* Remember last position in this iteration */ From patchwork Thu Feb 23 03:04:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13149806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D395C636D6 for ; Thu, 23 Feb 2023 03:05:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25A026B0087; Wed, 22 Feb 2023 22:05:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2091B6B0088; Wed, 22 Feb 2023 22:05:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A94B6B0089; Wed, 22 Feb 2023 22:05:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id ECCCA6B0087 for ; Wed, 22 Feb 2023 22:05:17 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C0B7881027 for ; Thu, 23 Feb 2023 03:05:17 +0000 (UTC) X-FDA: 80497065474.22.A11F957 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf21.hostedemail.com (Postfix) with ESMTP id F1C6F1C0015 for ; Thu, 23 Feb 2023 03:05:15 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=To5SpBeX; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677121516; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zvL3mMUiF6TzdAKKcnyDSGCQ6fTbHMRv58umlvVWhug=; b=7K1AE+PHN9NsoF1njII0v+D47fqzXfwORpFwZCrPh1AOcmMmqTqMiX2bUKJGLMfHhN6FNs +bCHk33eWlitooQm4JDdsVx7Fwei0Tab+Ja2U8TiyHQDYHpk52e62ofk7tRT2yd48f76P+ 3dQ/3pmEyIdVMW64jrFEjwc89x9k4BQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=To5SpBeX; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677121516; a=rsa-sha256; cv=none; b=RcxcauPtUUk0ufnMd3KszeVoa7Zwf6AzbRX/mv+V2nZVY1wNPAjfKnRavQgxkvMH/11zSW ZkRZuIlEWq1oGMhYGUTg34aGhta3dcBR2QhXlyai1tlFK9LyUupFVXt88oC5hbnQCTeQTT e4xWFQSQVgFCgFSacOhfWmeTj+5Elq4= Received: by mail-pj1-f50.google.com with SMTP id d1-20020a17090a3b0100b00229ca6a4636so11372458pjc.0 for ; Wed, 22 Feb 2023 19:05:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zvL3mMUiF6TzdAKKcnyDSGCQ6fTbHMRv58umlvVWhug=; b=To5SpBeXwNb1lYh6OtMcdA7YXbEuqfheT/FB+PL18Ki7W7SwW3kIj63xK421vOkft8 /Pk58+kjibDmqthKvSgleL1UrM3fGkIT+Ishh6Lj7spySCW5r10AOqyEFNCPUqfajtN/ z+lLB16S5+B2UtABxWkJ1FFeWDsRq0a98+46w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zvL3mMUiF6TzdAKKcnyDSGCQ6fTbHMRv58umlvVWhug=; b=PtQv61N1UGBK3Ojqo9BSTOgvuuFa1ZwWMzWmcEUTdAqjspto2ifNyQp3RBBcjK2q4P evCXVMEbTKfdk+/hFJW5Mxh45ZPPBaZfrp38/ZH4wVFme21NM50hnju1ZkoSVMiw/Dm8 VoPUtNBTcrQLliHvtj1dUy5tOg8wyQuaL6aLHBZSeABfSjPNs1WhoZ7RnqBLopyBbxqK bY7H9gbRr6Ka539va6baqNGMmZebQrrszr1fqIoOUNau9tEBmLUIyQNMcl/oEoeT5TpU ykA0GjTT8LNowDFGpiWG/vtRQw/R0HD5yre8WzW2OlPzvcJ4a+V84Kzt6wqAAQwi3OJv AYzA== X-Gm-Message-State: AO0yUKUN9gnOSGI1R7jlJA42ZhG8D2K3bWXE9qwgBLTU3fvvtTYw1VLa jwcvxejgvGtm+eDpMEXP00LQKA== X-Google-Smtp-Source: AK7set9GwcRpldktMmwsyDxV1U87D3miMVOwqhHkQYQzpgN7J9gWBZIAlMDN1gc6G/18tFmaOpi/mw== X-Received: by 2002:a17:90b:4a4d:b0:233:9fff:888e with SMTP id lb13-20020a17090b4a4d00b002339fff888emr11273419pjb.39.1677121514958; Wed, 22 Feb 2023 19:05:14 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6de2:9e85:b508:57b8]) by smtp.gmail.com with ESMTPSA id jl21-20020a170903135500b0019926c77577sm608520plb.90.2023.02.22.19.05.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 19:05:14 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv2 6/6] zram: show zsmalloc objs_moved stat in mm_stat Date: Thu, 23 Feb 2023 12:04:51 +0900 Message-Id: <20230223030451.543162-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog In-Reply-To: <20230223030451.543162-1-senozhatsky@chromium.org> References: <20230223030451.543162-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: F1C6F1C0015 X-Stat-Signature: c7py46b6yewrm7jchigj5n5akztqzfb1 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677121515-454381 X-HE-Meta: U2FsdGVkX1+4HYQP4sE7HEUmQOteQ0u57bgf/3/l4VnsQQNkgkQGlZ3ugAugwqtDiocPPgZ/c03grG2ODgPAAOATn8QSH7v8SYIGXFYSMIrNc9xUzACg8cQ6hiRW4GRggLfUToD2S0JB74YkKvJqMZmkWXcVvkEUY6SenNdSi8+e17sHATNfiSKcXnKfHFANyb3uqgVm7NcrftcRVRuHM+iiMm2eookPpJLBr6FaLVurk5eszj3IJ/k+H842lck6UF+iUA8ROpAlK++PM6c3XAGIEKsJYirFqrFC/1rRJRp0KvggnsME4fhfksf35UPAtdE181AQ/Ya5I0Z/CqddRfufwzqwvuH7MKen13eoBssTxdoHY0vpzX1uqD42S7QEFEWkqsCU+6lSi70xhHLGNGY/A4+rDUv///YbIoTp+zt4V7mdKLt8vGdmaC1PI6THoqgznYJlVy95AW8nByJUpAuE1MRKTXRFFGsl5srAm8p8p97Qg35UUhXYqaJzUB/jKGyN1vIMKVz+kzl7A510RPxlEMbvaMtwL/RqUv07zvfMUTeZxrDhRe4WhcDr9eN4mHWGl+ZwNoFHopzVoOW6xW2W6zYxSJPrhPtTCpovV9EN6XaevKzugj/R7KH/R/L2q7G+QaKThoymveR/7tCl8AzLYLiNPevzinaB94TM+PrQNYaHkYSxA9kDpxGwG0IpsJ91lCKvT2dQzzmxSqE/exXivIEFjURkuzhGzoo5/PUzLH0JwGwSOsrcW/XML+Y8rJpIMDGPBXDAW2UOczzx7PkTQCVJRwKHNfUQkmDxu9N900ujANE+M2nvGA+SWEZ3VECvHRDeyii8tIi9CszRpRVgfFYO2p2S85uE+hYgJTqsU/MxxD1CkTihrP5vWOsYw/abPxUfOAtUmkPiquWXDQczvxsejvWrJwqB1yVC7nW280r/u4c1scQ06sAvyGmKT94RrSyO/sX88ioDDjl r2d9JcLB gtG6Cb1iqV48Fji0dgn8fRE3uwC9jIehSvNeVfQ0lv5sO2imDPCWrsotoAMvY18FBf6wZHBaFKmhrfFyoe4eXRYARMyOWKOC/CfnZPUan4WBne0jkaYvrf3si4V1So6Gl7izinJkzpQ/gzGfkdYAF0J0HpYs3CvWW7JLTf6tnENwDdLKw6h9ZaDPffRCvL/DfFAnpq2nvH3ixyn2ZDtYNfYzIB50df9I4cEveBBSB9ZtLDBmGcxYHO1/+V4KTMMWSM3zfJBpO4wgFMM9SA/sZkONTEOy9A4TRxdb6Ttr1E4deXBZo+XHWfcw+ViXIhwT+7vEZSxBgR/pNbcp5kjruiUxhB93FXQJyXrCDQdk/J+Eop017hnAklI7SjivG5AM3FcL5nds0mbpV6L91HEvRbef4FNVrqF7DpHo2d8xQunkmlPQh9b22iK4ocbV7skjkMTUL X-Bogosity: Ham, tests=bogofilter, spamicity=0.001536, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Extend zram mm_show with new objs_moved zs_pool_stats. Signed-off-by: Sergey Senozhatsky --- Documentation/admin-guide/blockdev/zram.rst | 1 + drivers/block/zram/zram_drv.c | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/admin-guide/blockdev/zram.rst index e4551579cb12..699cdbf27e37 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -267,6 +267,7 @@ line of text and contains the following stats separated by whitespace: pages_compacted the number of pages freed during compaction huge_pages the number of incompressible pages huge_pages_since the number of incompressible pages since zram set up + objs_moved The number of objects moved during pool compaction ================ ============================================================= File /sys/block/zram/bd_stat diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index aa490da3cef2..3194e9254c6f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1221,7 +1221,7 @@ static ssize_t mm_stat_show(struct device *dev, max_used = atomic_long_read(&zram->stats.max_used_pages); ret = scnprintf(buf, PAGE_SIZE, - "%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu %8llu\n", + "%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu %8llu %8llu\n", orig_size << PAGE_SHIFT, (u64)atomic64_read(&zram->stats.compr_data_size), mem_used << PAGE_SHIFT, @@ -1230,7 +1230,8 @@ static ssize_t mm_stat_show(struct device *dev, (u64)atomic64_read(&zram->stats.same_pages), atomic_long_read(&pool_stats.pages_compacted), (u64)atomic64_read(&zram->stats.huge_pages), - (u64)atomic64_read(&zram->stats.huge_pages_since)); + (u64)atomic64_read(&zram->stats.huge_pages_since), + (u64)atomic64_read(&pool_stats.objs_moved)); up_read(&zram->init_lock); return ret;