From patchwork Wed Feb 12 06:27:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91E48C0219E for ; Wed, 12 Feb 2025 06:33:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1782328000A; Wed, 12 Feb 2025 01:33:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1274D280009; Wed, 12 Feb 2025 01:33:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2C028000A; Wed, 12 Feb 2025 01:33:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C9FDF280009 for ; Wed, 12 Feb 2025 01:33:00 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 89886B14B7 for ; Wed, 12 Feb 2025 06:33:00 +0000 (UTC) X-FDA: 83110324920.03.6F05DDC Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf29.hostedemail.com (Postfix) with ESMTP id B0AB9120002 for ; Wed, 12 Feb 2025 06:32:58 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=jK1qRgHX; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341978; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=icrLtmzu/aeIpX93BXjQOW1a7S8CNx3CQ5jcnIARFMNehzdk6vH3nPVwiF8pBnAUyKEutI YT5iP+Ko2nP4E/jsJkceH9F4RFI5CVN8kE0CvP47H/THjyEiT002OpKEyB/5vFMlz2VDme G722zgQE1VksOa9XFWGpTZz332sKLug= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=jK1qRgHX; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341978; a=rsa-sha256; cv=none; b=vqAFRkpI1SjnBG1Io6rIIoPSuKbd8oV91xrVCmuWv1v8dCqhQXKgsgU+5PfFHW/Y71pTdZ 0rwwp7ahDGvghUlW31QcBz/V0y1KkvQCGBJWohx6OgPfuiRrRTicFp6yaOGevmwuSEwF4Q i3d9E8K1jON8+fQGY886O7FvvpazTYw= Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2fa2eb7eb45so779063a91.1 for ; Tue, 11 Feb 2025 22:32:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341978; x=1739946778; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=jK1qRgHXDSkdsnMP+NxFmESyAntb3bZvpbcDzqM5xYe40Tzkh8tFPtjpasj8AvvMJh GPtiQ9lWKjhpPp1Fj1795XpnHaSgJcilUWOIS/Z0eEQEM4z3BuR0j+EVJtNtVPY8friq 5o5LUJVI0wVE87l4IiA+W2+uw0AfPqOCQOW9I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341978; x=1739946778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=IdrthtHg8Eob+bCdXbNjxHxSF4pu8PgifQWdccxxnk9seFrhEReqIurml2JdFFroNn fNhF/fEy9Ffhpucrt0UD2zOOqCYHnKj119oHoJQumjYJAvssBReBV9W6l7yhk5mM5OQW gQSSKiEHES5cODJ7hjFb3DjL9PjigPDZfBo3Qof3tCodIUlWs9Vqq/fpad8zOl81ZK2f DlbV2s9sFwdmOT3Zh6M2KFMcf1j4gUXEGTXC9L2mjE87lUiCunhQ5UCjdJ8cqCUVI1ox GzJLOLRYybxf5oGiAoMeKAphP2beCPES1trOwdjGcaLisW2WzIUme/LpEEvpuaLLpWIo SPjw== X-Forwarded-Encrypted: i=1; AJvYcCW9yyx8VpYENOQKxfNkeMeVKFo9hptzSosa4rG0T03ELgXddlO/o5pZr4ajBqBeHI8OtiEZYq4H+Q==@kvack.org X-Gm-Message-State: AOJu0Yx83CjstUrPHxnLkpS53rwc/l0988jyeWXwBE6R7u8YYbsDjts5 tFj0Zlq6fb8bOWgWQ3RyCiXxgMM92joJ6JGTjVY23YNrMVKMPh14xe128kK23g== X-Gm-Gg: ASbGncur+NzikPO2bNcZJGmntUDT7dDqlXKT8VAEXM9GCiVAJtO88lOHtXTSZl5rsVO kPX8Z1oLbNg2wVeoPOwC9oPyYWlESugqVC2bcS7/hkapXUPHv0X02wgr2VCRozqMi8NstC6nVCA F0urS3hTthxnRYlHZs085AQwPdUaXNERf6GqLLQxcHUwL2simt5LYLThZeAGIPNbflzgb+B3NM9 1Mo4yd1zmgRTYKi+yG2HJkDlDvNaKf64ICJaDUD3KdzlqPDWb5J9bHHD6FkrApqNwb5IPSvCGTS lvJsTkwh6+fOiP82Lw== X-Google-Smtp-Source: AGHT+IH9novtJAP2nahMJtf23e6d5DXnQj2BtTSpA3K6387kmaUUk7tqOB+00czVkoYIJ9iJRKWKIQ== X-Received: by 2002:a17:90b:2b45:b0:2fa:1c09:3cee with SMTP id 98e67ed59e1d1-2faa0913acemr9497572a91.9.1739341977607; Tue, 11 Feb 2025 22:32:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fbf98cffddsm658335a91.21.2025.02.11.22.32.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:57 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 11/18] zsmalloc: factor out size-class locking helpers Date: Wed, 12 Feb 2025 15:27:09 +0900 Message-ID: <20250212063153.179231-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B0AB9120002 X-Stat-Signature: 31ydqnfpy1moo373srzrgcwt5nhy8peh X-Rspam-User: X-HE-Tag: 1739341978-988025 X-HE-Meta: U2FsdGVkX18ggElPVKeLecKoRxP4kAyJrcyDe90awrdznR9crdTQd6dYRjRl+IuC3TnKX9P0qIpmZ84tZsByxGWk8MzwpVMK1h97JZ/Q/JfkeMUT2QFy2Iad//0HITy9Vgudynd84RaxwgS1Kc3jwCbIAsCW++w6SBEzXXGuoOB7RlY8ZBi7gPNpwjTRIZfzGE+LqcYcgO+OIlc8r3T5xwYXpDiKLT5L1ARx7EPGTI3FwwSbHV9p2iPR1FoBwqswGrEz++R+XHqXKtpL3RW0cy6YEmQPCuz7zvMcp/tEZNHTVpsTUdrnSmM810piKRCX3ey02i1CS0YsXnQx7v+C3Cu9RdRlEz9ztDAbHCDvENMAYGyqvnAGOHXi8cQl3TzdFacZBPxgk14rXB1cRFPPL2zWi8vLCMpV2hyO7Ejnrl9YNRCfPclpSGksNqgrRJmUsxVE+cO7wQB1F+0ecuGoZCFPJYg8D3GcuNR5cZlATSSNaCDQyGswyyENsqUwqDB5PpYqJtifnVIWeENpjNppc/7GE0FsNFD8OaCbtnB9z3zZ97Yzl6JzvICFsxpiy6dVAllJ4Rl4Zb25kwIUgRt0j11rjeOSebNV4knm9qeO6B21S5JdJCf8Suy85Obf02Z/Q56K4059KT50atCwxSCHj6aA+I349s9FD096fqy/IX1nb88VD8tc3CPAou5NZw6tSbS6cYnuFtdOqGoeIu0yIMGnF5n8nyx0j/BUCbyRe566ywWTJdRX2sQYFqP9WGB0ZWF9s+S2kXfbivDCVOZWCxP/sT3F+QdTujSsuLLNa8UtqUiT5XvIEFHneLPkP/B5V/f0mnd56QoDAq5jQe1xFyLNR1j0ZJd4vZt3VWA3EAmaNR1aBQNYMhG01oklyyBDyCFqEFxH3UoUQnrFg9LfAKG6F3YFAquFejlU1XwLioq0XI3QersbtG7X2i/vWQnccBrgKC1RDZ924cDGIzv QnPEHr7s 9GKxWB/P1GceCOJTwjJUhmwEHkkwyYFJVQO84BGTPTU/oFrv5u2OzgQAcUgVPMnt87fN9WzB2SzLD1wYUZpNTLIMaXVknm0XQ4eXiQpeBSlB5KSFYXsX4Wx/Gwy32CCuG5yl6MebuNLi7m/KmGE5ZhNbnjG6W7goY+xcJ8tDhyqGN7nffagy6Z4zRNtr2OXKyXKwC7fGI8ymunf9inDR6Qw6XsqbyZq5a2ZN+Rx889ImbGQ3PLNIhPjOTUnbMFSLuBU04aEw2LQiMIp2yDiGQspv4xrskgAdobRADODP92c1sH5MoxLqUNOUbIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 47c638df47c5..c82c24b8e6a4 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -253,6 +253,16 @@ static bool pool_lock_is_contended(struct zs_pool *pool) return rwlock_is_contended(&pool->lock); } +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -613,8 +623,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] += class_stat_read(class, fg); @@ -624,7 +633,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) obj_allocated = class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used = class_stat_read(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -1399,7 +1408,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) class = pool->size_class[get_size_class_index(size)]; /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage = find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1410,7 +1419,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) goto out; } - spin_unlock(&class->lock); + size_class_unlock(class); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1418,7 +1427,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1429,7 +1438,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); return handle; } @@ -1483,7 +1492,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1493,7 +1502,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1827,7 +1836,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); @@ -1859,7 +1868,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); zpdesc_get(newzpdesc); @@ -1903,10 +1912,10 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1914,10 +1923,10 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; @@ -1982,7 +1991,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; @@ -2012,11 +2021,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); dst_zspage = NULL; - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } @@ -2026,7 +2035,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (dst_zspage) putback_zspage(class, dst_zspage); - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); return pages_freed;