From patchwork Wed May 17 08:58:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13244469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FA9CC77B7F for ; Wed, 17 May 2023 08:59:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BD75900006; Wed, 17 May 2023 04:59:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19EDE280001; Wed, 17 May 2023 04:59:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05C8B90000A; Wed, 17 May 2023 04:59:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EC450900006 for ; Wed, 17 May 2023 04:59:06 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A28B5140485 for ; Wed, 17 May 2023 08:59:06 +0000 (UTC) X-FDA: 80799147492.16.4A826F7 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf26.hostedemail.com (Postfix) with ESMTP id CBC2F140014 for ; Wed, 17 May 2023 08:59:04 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AhqU7dUp; spf=pass (imf26.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684313944; a=rsa-sha256; cv=none; b=e2zneVfdPCzl+KsiDVDiUTiyU9IRNtXsVol4XVXAyXLpq6WF2tXDhVg8o/XaAlh17Ly70Z zuTgU13Wd3s9+Oi8wU7OS9tjTARfLpFsxRjbDUEmcXHrO/yc6hfFBGMBLBVjmTI7/7aiJr 8QflbJsQ+HFbjOpZt5OO3m4xviTJzFM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AhqU7dUp; spf=pass (imf26.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684313944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sATgEzrR7h2S265Iki4mbf6UDAayTpXJc+pupXreNbo=; b=QL142KBxEhYBoKTerbaKOkmbWK0+y7VMD88qLQ5nSATEiSYM8dMOxCODxpnLIaKTGJQ0wJ Y4DfyFy1ZOSbRclvQc4Zr7oOX6eCypDZpbW/2rtWOx0G1PX5l3/VsurNvvUm9S+XxSGqrl aYLUFFevlrgk6J8/+Hqlb9ZXpFOaD10= Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-51b33c72686so325494a12.1 for ; Wed, 17 May 2023 01:59:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1684313943; x=1686905943; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sATgEzrR7h2S265Iki4mbf6UDAayTpXJc+pupXreNbo=; b=AhqU7dUps8kowzB+osQO8so6VQf0rmRE6E4zIjuNyIPM6ywkBKH1KWI+hwH6Hb8grH bCqQ1C89pxuZN6S/Qe2mlxhm/jIwJYn47SJ10Pr+CL6qSbZU4B9m8pibG8gDHXwaGqha BVt/etviuKp+jgzPJWFF1XHYRdgXwLWhKdMu/EIsq8iNLQAuAxCx+XmuV8f4Yk++dfHM zNgtI4hXPq6uzaYEDuOkU3GWF4osZuQlgfHh6H/lds1CnKY1wfGyfMt8sEPGBvtX5fXQ FZNAo6CIYaoCnSAF+GU3yXvBnGH5ReKVI1SptmJCddaenxpFVlVGwF3/6ZCTN1QOOPcH AItg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684313943; x=1686905943; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sATgEzrR7h2S265Iki4mbf6UDAayTpXJc+pupXreNbo=; b=LYCudZBJtOhzvVLMPzaeUef0GbQALGTU1zUpNv5t/1hsWZ9IPu5sRwpRafYbpQEwgw I+IduV7XNuPg6Pqbowp6Q9QXA4lbZ7k7b9RkPW4TSRjk7vC2eU29jJZ+/FnmL4rowiZX ctkJz4JImbp7Rv0Bj1BSkKeNhZG9BVsASXWxucR5uGJuH8SaluUbRQgsqmm6WOdiwqRO q738NMK/gDdlnGCc9AgHIxnLHzYxKOXZ+ve8WZWFcUuJRXq4873HDIVUDQ/g+Lzy4fU2 E3OFY4DbnN9VhQxlUGxWCmpYulBXBKn/vvSSRfq0JOZ9h2qGJqt8iZW+kaPW9K0Pw399 RSdA== X-Gm-Message-State: AC+VfDz8c0d0tNevmZpEhIljMSyEoIo4cQSHnt3dKl/ahHs00rjL6UTM T7GHmxQOxJUA+7E74OcVr23T+g== X-Google-Smtp-Source: ACHHUZ5pxBoy9i804S0WFEBY4chpbe7G9WMqboedp6zXSUYx96x0ZmEDxKvpxQw+xImuE9quMaS2dQ== X-Received: by 2002:a05:6a21:3298:b0:103:6c36:46db with SMTP id yt24-20020a056a21329800b001036c3646dbmr32420717pzb.55.1684313943206; Wed, 17 May 2023 01:59:03 -0700 (PDT) Received: from GL4FX4PXWL.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id m18-20020a63ed52000000b0052c9d1533b6sm15056765pgk.56.2023.05.17.01.59.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 17 May 2023 01:59:03 -0700 (PDT) From: Peng Zhang To: Liam.Howlett@oracle.com Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, Peng Zhang Subject: [PATCH v2 06/10] maple_tree: Add mas_wr_new_end() to calculate new_end accurately Date: Wed, 17 May 2023 16:58:05 +0800 Message-Id: <20230517085809.86522-7-zhangpeng.00@bytedance.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20230517085809.86522-1-zhangpeng.00@bytedance.com> References: <20230517085809.86522-1-zhangpeng.00@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: CBC2F140014 X-Stat-Signature: ippietrhgurgwcaszribudytuxgo64gk X-HE-Tag: 1684313944-428518 X-HE-Meta: U2FsdGVkX19KE0hHwgSFqATCUk3QjP1p7z6pbD/L9DNUPM3ay0gWsZWt6J3lLySjQqe/qbrKBPgdNZFXn2KwAiTZHMT6Gp/VArTWEsa8y5DUSsnXhS2ecwAmlyglElHrmAIwwz9wKVouT3qfQfiihNgc0PxQBYSosbOcUO0QoHVmD5gqRO6H2lOhO7w+RvCpBm719H5WftqqHst/ZJQdu4EJbP/wdMkauJ9miQWx4ly6kIb2+Jfk6YRcPvCv/eGaE5xIjxx+Mx2rbqq7c02539+DXuI4qOFblqVsMndUarTpCPYK51smgVq12J2RZqdQYgvsNywNyZ+OIM8TWjn3WUzPxdP4m3YImjVP5FCsU+Yi6TNIvLYaNbc6sAP8LHCNldfa840kkhvkh009faQfWC6MGB4N6L90MXwL7uwUzuoD3j+LJuOf4yVgTXy3FpVPnNnX7ofx6O9BRTtAsDyYSVDi2Tf6HaLdXYLsBnlHB7/j242v++47EHxQWV7VlURfV6rxx7hqldALxx3UpVv7L88z2JWrg0UV3uASe4+tm8YdDfJ4n1LdYXNvZyWKcEisxhKwcZ0z4PMeVCdEpYzoJTf3y4Edb4DEeW5BVY88f8fMrb6AFWot3yI8LT5HElABrXTzUl3U2+8H72pIMmY72kAPF13E4k5sSTn29odavc7E1w0IAykdV5wzwprv9lcPxq3xcGyFFU3qLg6V7p4jZOPrUQQD6uvwbuaOWLZ3AC4UTtbol0/nhmxtLvsDeMle+EvzhkMmFSq/+hNPIU+fX2V+X3oUfVNa0VDQeFRwvnZGeNV5hKErXN/ilKlwelv9DZWDx2bAtbYmcBYyEgUdinDQlU3FoMzlNrLqJPfoMDAwmUdO8D1CIvYLUca4Qm6M3bAUPVlrINSDei/2iIf/lpf0gHn9AoxBPVtqqYC4FXQjMwJl8BZd1KAkRaRcSpgr/F8s/S5B52Y58MMazKd f4uFN8W7 PGOL4SEPw/1WkZ5jfUBYiZWCDLIgDmdvOPPa/sa+B++BI1v56P232HxFqImf2OnKx4LDV9bWgdb8QUa9NcFiTVhvjD6kc4JEY/JXEjt93k1KydK/YoRU55vlqxIGwtc3W1hbfAuiA0RewE/koyY5/ou5+/RVYOytDz4b5ELqWnGttsQYywlCZ95zH28AlBUe5fbXefWAddo+mLLZKLkjQ4Qdg4XGwGzaYgr8SvALo82U3OY2K7G07kEDtzLyxvEhymVfjSjiH5gjJtHPUFi3IAQLq7J8bYlHvaRkkVF54S+G7ol8slNitIhYLI0ze5MddcgODsNCCnI3gdIsF64ySRE5tR4QbJa4pH+QWDGmee71pD+Lj84GPncUR30mfNuZLvKTDEcZJu3VA2K76E4BksUiJmrybvZaC2s9RhIQxa/6Uc4tb/ip7T8i17YNkEm0aaFBVroyfwRPY+2E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous new_end calculation is inaccurate, because it assumes that two new pivots must be added (this is inaccurate), and sometimes it will miss the fast path and enter the slow path. Add mas_wr_new_end() to accurately calculate new_end to make the conditions for entering the fast path more accurate. Signed-off-by: Peng Zhang --- lib/maple_tree.c | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index f881bce1a9f6..3b9d227f3d7d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -4294,6 +4294,20 @@ static inline void mas_wr_extend_null(struct ma_wr_state *wr_mas) } } +static inline unsigned char mas_wr_new_end(struct ma_wr_state *wr_mas) +{ + struct ma_state *mas = wr_mas->mas; + unsigned char new_end = wr_mas->node_end + 2; + + new_end -= wr_mas->offset_end - mas->offset; + if (wr_mas->r_min == mas->index) + new_end--; + if (wr_mas->end_piv == mas->last) + new_end--; + + return new_end; +} + static inline bool mas_wr_append(struct ma_wr_state *wr_mas) { unsigned char end = wr_mas->node_end; @@ -4349,9 +4363,8 @@ static void mas_wr_bnode(struct ma_wr_state *wr_mas) static inline void mas_wr_modify(struct ma_wr_state *wr_mas) { - unsigned char node_slots; - unsigned char node_size; struct ma_state *mas = wr_mas->mas; + unsigned char new_end; /* Direct replacement */ if (wr_mas->r_min == mas->index && wr_mas->r_max == mas->last) { @@ -4361,17 +4374,15 @@ static inline void mas_wr_modify(struct ma_wr_state *wr_mas) return; } - /* Attempt to append */ - node_slots = mt_slots[wr_mas->type]; - node_size = wr_mas->node_end - wr_mas->offset_end + mas->offset + 2; - if (mas->max == ULONG_MAX) - node_size++; - - /* slot and node store will not fit, go to the slow path */ - if (unlikely(node_size >= node_slots)) + /* + * new_end exceeds the size of the maple node and cannot enter the fast + * path. + */ + new_end = mas_wr_new_end(wr_mas); + if (new_end >= mt_slots[wr_mas->type]) goto slow_path; - if (wr_mas->entry && (wr_mas->node_end < node_slots - 1) && + if (wr_mas->entry && (wr_mas->node_end < mt_slots[wr_mas->type] - 1) && (mas->offset == wr_mas->node_end) && mas_wr_append(wr_mas)) { if (!wr_mas->content || !wr_mas->entry) mas_update_gap(mas);