From patchwork Fri Jan 31 09:06:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 356FDC0218D for ; Fri, 31 Jan 2025 09:08:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B991A280290; Fri, 31 Jan 2025 04:08:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B486F2800EF; Fri, 31 Jan 2025 04:08:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 99FA7280290; Fri, 31 Jan 2025 04:08:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 762CD2800EF for ; Fri, 31 Jan 2025 04:08:02 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2975E80945 for ; Fri, 31 Jan 2025 09:08:02 +0000 (UTC) X-FDA: 83067170004.08.CAFB8AD Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf26.hostedemail.com (Postfix) with ESMTP id 3927514000D for ; Fri, 31 Jan 2025 09:08:00 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hkpZUgGo; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314480; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=7ttbmv0/AL1oeQ/dCly+ig/dJImNfSb/EA9dijsaptrQkc9AYKcEdELYzbVDm6bZmEUngx gVqFC9uuuI/Ubk1prquMWfFRGo+Wduypau7BKicWzSG8jcEWoNFUmwdO3E11PKDKccqmtK Qv1JrBsOlM4kpE/xvoagOUN80qhw+XY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314480; a=rsa-sha256; cv=none; b=BnWCgTgs95nwBwK2CAzhQ59M6OHvuoirBS2dbjJfdzfEKDqM7i4xKJGOlMrBbi3lOqeWjk khMqPOFf1CdZUbz6huBzUBInmafUVSaBxH6DYHJX1kZTjUBkr2tf9TtdsX3GI8GuVrCP/O oO8s1ngUQj2SEPE5oHzAxyHYCOFy+/o= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hkpZUgGo; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2163dc5155fso30739575ad.0 for ; Fri, 31 Jan 2025 01:07:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314479; x=1738919279; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=hkpZUgGogK16hR4UUA7ncqGEIw1+BAl5BxtGDJDzSqRJvLV0IH2xj6LAl4UXBFqjSC gZCt5WrrqlorGkY8e3reeBojdBlqI+KG8LRJ6L5jOYodv6oPxUm66NGdu0oVlcpNGbR4 2DyzGw/rxdPadaaSiq37kTXu5kLOdtw5kMjrg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314479; x=1738919279; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=shZzHw4ardfq/JHKTVCioKDkzuJOn9HUEIfy4+ToWX5DaZWNfnZ8RFgUbKrwHo0KBo 2yGVD5hx0CH9lO/xv8IHyZqE60wbDY4zW/+/QVzbtQwlSEF+X2g0bmfZu3QCwR4GwUfu BmJOxRh2iFayyK6lm3mjAI2fNDZmx7s/6cn4C754xn3Nrr5OyV1/6OxVpw8J34ip8/ec 0GemAjuBq8S7qlPHMswSGt3ADNzn01+sOqvac3O6dyqZuz13D/tT+hrwH1VSWLUpBr9y FQf7/yoVpfFybk8nf5IDeptZGBKcEhCyddRJOpzVANhHkVQDtWt6G8F47wyonk60tXhD i1jA== X-Forwarded-Encrypted: i=1; AJvYcCX78l4jYt2yDZ3V2ZQ2lvyCaCAni8D0d3PIFvLpxO8pYp5h0KDsrNd2J8nqScre0MIkkuoXGx2PHA==@kvack.org X-Gm-Message-State: AOJu0Yw5Sel3biOeKx8rY6tdNUJRDUBax9EolR7PkY7U1W2LqcWZV5Pm yFYq428WEsCrKs8eBy8daDkRCuH5Dah7P2lSOpwQL8w9l26PU71xxn369D1ABA== X-Gm-Gg: ASbGncvdxuJrzy/lJpSBSqQADcEI+XmZC5hdrht6VPW/DZsG7dYhbSktXR6LkZg2dHx sIooIY5rmNoq5uC0oRcGlsQWizhmW3Eq69Ei53euLnfg8qDG5jHMdAA7zuDdHecIWDWi+e0oNzC rO/GX3tF5dksMECb8upKipTZ2g6285/4DTVtMhcERe6Ie7YBH5df7GK+8D0mDcjZQv4tejuQhWv RrFV0r/zyoICCGoZxarmRH7i6rPqo5cP9TAqOF/JW8MRETG2TgpRd3eUmzoBp8nLo6x+6UlzFPl 9QgGgrUN3HD82XQNZw== X-Google-Smtp-Source: AGHT+IHhqjztMQY9nyoxNY57ZCZdxtWzFwxbZ4LRhwZuQGquqZr7M2SqvkrDWTSh2oSFv8q0NO6MeA== X-Received: by 2002:a17:902:ea06:b0:215:352c:af73 with SMTP id d9443c01a7336-21dd7c57eb0mr137738715ad.18.1738314478999; Fri, 31 Jan 2025 01:07:58 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31f5a60sm25837715ad.61.2025.01.31.01.07.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:58 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 13/17] zsmalloc: factor out size-class locking helpers Date: Fri, 31 Jan 2025 18:06:12 +0900 Message-ID: <20250131090658.3386285-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Stat-Signature: ycbta3n6pa4k7i4594zze7h9r1g4huna X-Rspam-User: X-Rspamd-Queue-Id: 3927514000D X-Rspamd-Server: rspam03 X-HE-Tag: 1738314480-758529 X-HE-Meta: U2FsdGVkX1+ftu26S1pP2/ogM0FKner4poYvGqlUIgrg02ne1UM0ZT2DAIuBys67E0/VC1JcNuoOFb5mMLJkgpv3H/o9DiObawLMvJBcIZNQ8vPZm0P988xAfCSUc5FOjfH0ySmpyjl3DvWyE3MUwzJcPJ2nT0z3BnBEqguu1NERW4Vrujta3psTHhcZZ45f8IWl70c9IfKYay4g3xHr0l4VbqsrDlwYwmoDwpqe1fpBiA2D6sGuebGRkitcs5tHq1Tu5MXqt760g1uJBpINlwzXozEVzDtQ9AQf7ZZQ2ewoMJMWkKrWcxtrtoloJAi41KZJVMCnEiVQc/tfjhAIb81Ogh7tlfAytB6vPNKI2YsVozJRIzCkiwfxk039vR6BzpBOy8Xt2vpyRb3TPyovqVj4yPKpbmf5rd6h2r7e6WArsj5YrYYCXaCjQXxy9GRBOKFjDQ1Hk+mqK4vuSD9OyLpf9s4Dqf6C1j3BtJu2yLfe/V1O8tWp1+H4pYl9z2vUTbjNRtDeBUnLIGLOQflsUavyaGe3SDm8itH5Di3wjVbUnbJgL6tR4oiWR50LWwR/PCydyBLN5t8DLR0gfE5OIycN23H8jiM+n/ZLf8+6/RVs2/H6FAWy0L5NEXUtpIQL1o0yukpLmA4j9lFfgSn110fi0sJxL2Q+Tot1J4jenfwf2P7HSjESGYdA2OUYDc3xyARCMT2PhxUc/cCciihLq3IUn2hOH7XaZqcS7a3V8e9KoMZ2ueZDwjSWJCQblmIPdXnq/q5N2nEDuoQyj4b2nQued467xIhq1ZH1TmOEtBwzpDl/vwB77GtXHFV2k0Q0aXer0Xdax9LC326v4NWXQa6dvUEYUja8uyPai2JCBZ60VwmJpMI2XDRDO3IOqRdhyP7KYPdCa3pR0gd6t0j8ghvICP7PW5ezQfw8ajh2WeXTza0C5YCqcQD+Y0f7yjKDKtYeq/fIc87h5Kvg35x INlILLhE ybv4Zh4KsjMLS4MofJT3Ut+O4oRNQ5IkgBDNeqyAwBk6DwhtLxZmeGouh+xAZOVC/cDc43VpwuDEf8/kglSTjOXk/N/nVEhoBCrVNGddiLInxmEAe9cOVX3uMSnmPw8PMt8+vg3uDSTfZa1PAGXqgOqWvO0jP3F/rW5NOCcy4dalxCJMbVuuveGoEaIdZNGADQ/4GKQZycOBv66WmwLNR7ahFttQRL0jeixzuw7CvGWVf91B+I2mCH/u/1K9TDPKTr5NEEkMCjTV1AErrXHDt0WH48P1DpQb05bd44g6uwvCh9mWoSr77gJZINQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index c129596ab960..4b4c77bc08f9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -253,6 +253,16 @@ static bool pool_lock_is_contended(struct zs_pool *pool) return rwlock_is_contended(&pool->lock); } +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -613,8 +623,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] += class_stat_read(class, fg); @@ -624,7 +633,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) obj_allocated = class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used = class_stat_read(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -1399,7 +1408,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) class = pool->size_class[get_size_class_index(size)]; /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage = find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1410,7 +1419,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) goto out; } - spin_unlock(&class->lock); + size_class_unlock(class); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1418,7 +1427,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1429,7 +1438,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); return handle; } @@ -1483,7 +1492,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1493,7 +1502,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1827,7 +1836,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); @@ -1859,7 +1868,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); zpdesc_get(newzpdesc); @@ -1903,10 +1912,10 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1914,10 +1923,10 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; @@ -1982,7 +1991,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; @@ -2012,11 +2021,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); dst_zspage = NULL; - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } @@ -2026,7 +2035,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (dst_zspage) putback_zspage(class, dst_zspage); - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); return pages_freed;