From patchwork Wed Jan 22 05:57:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E0D7C02181 for ; Wed, 22 Jan 2025 05:59:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A640E6B0082; Wed, 22 Jan 2025 00:59:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A13D16B0083; Wed, 22 Jan 2025 00:59:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DB986B0085; Wed, 22 Jan 2025 00:59:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6BAF46B0082 for ; Wed, 22 Jan 2025 00:59:09 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9F0AD1A035A for ; Wed, 22 Jan 2025 05:59:08 +0000 (UTC) X-FDA: 83034034776.03.EC3A4B6 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf13.hostedemail.com (Postfix) with ESMTP id CF93020006 for ; Wed, 22 Jan 2025 05:59:06 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=VTy7NYcf; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525546; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=bCvFdgt+jBDfPOYmZNy4prZ6R+d2CTDkQC0MQJxePrA=; b=c2yQxLAEhwj8+h2Tfj9rvvLjgACaCeYAcl4MQyLmhc7dccoaCBpegiEQAZOETU/hgAC6Qc Ow0Eet2lqs+ei4OtSx8AZPjfype2jIPdFeqOQj16BiTV97ZI4hPw6iJ08GCQyse0pmsUWG Y3XvyPrXbzbJU8NofuVTqRRGxg4del4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525546; a=rsa-sha256; cv=none; b=RJoAmLlHrBgPZPqa3sOjYLse2Y0IVqfVBgQq1gcCFv61k1nuBufzRdN1OWCtQts1JB4eOO XI0ExqWd3qDLLt0OkQhC0dRiOMAKPnY51ABqMfcnFtiyD+/X/VFSbzK21E8lwJzvinPzzw ianGcBMPf5oKqjXNFgLpfqH6FVSiyzo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=VTy7NYcf; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21654fdd5daso111808555ad.1 for ; Tue, 21 Jan 2025 21:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525545; x=1738130345; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=bCvFdgt+jBDfPOYmZNy4prZ6R+d2CTDkQC0MQJxePrA=; b=VTy7NYcf38fATbmzxgtDXe6qOObWDxCFhixx1dpIiyyUG6UHXMi9YEwN1QZjvHP1zq jQQjqoInGEpb5yr89FaYpsJZEpQDdrH7ABcSTdI3eVX+p8Th4Q/TUxoGIxq9Cygjg0y2 qjtYB546N1M5AqDVkFqShtKi8KojuYnIT7hXE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525545; x=1738130345; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bCvFdgt+jBDfPOYmZNy4prZ6R+d2CTDkQC0MQJxePrA=; b=XcptfxxMw3PGV9s3t562BGPtPw6/452noDT1R2n1yh3NrGBgeqfrrs+t9yx1+MI2/v 2nvHUYyh1Mt+4T8OL/4vVK+zqeRoRhHPKgzsavD/luHdQwpuipG8iuJDk4yxs3RQFOvm hXpcOZul2bpjeqwGzkXpwlDPspELeB5kABaKHdipxz1IthWmRLsTCFbaVqpBxeQHrRie PtivTK9xGFX5o/ODGz1rqjpOWj3Q4/YRbzWf47dkpw29hQLQoglkR7+/WrGgpu2kL/oI He5bPuKNILv5+Ob0d45OgnQV3Vvg/JGLX9RNCMrGeDNFWJVE2Xaa86GdJmjqpr6H3/ot I7Hg== X-Gm-Message-State: AOJu0YwabsM6o1/+v23jcy55ybrEdr0WqDrkplpK7k0+4V5K1REAMLaq 3Sv3blHbGVpEUu8DVvylsZAyeC4FTM61j0ERO0l26i5UhQIXeMaKH01NMy98Uw== X-Gm-Gg: ASbGncs0DrrydS2AclXWJOVRjAyq0PYzpEtUAHLLQxxBjHjnpYcQXXB2uHrSlOiVcAA 6qdx2PxdfKDSYbTuTq8EU+FR0NawBlumTfWsFhqjjHTElS9JrrIHhZZRuxMTwLIMSpm68WUa9ef bJq2wGu6+x24QXwO0XBbBTPhL9sgQDPDmzSd8SKunUS5a/Wb6iChvQuYof9Z7iKFLfo+kOCLog+ NvmrMNBBfxbW5bFL3a9JY6Wht/8k27Wd59Jth2Tql22FxmI4KSxLIe6Cv/GpizpuB+PjwB3 X-Google-Smtp-Source: AGHT+IGAhdUAYOQVYlgwcV1bB8epb0cMowPOD12KFYTGVQV7Fx9gwBZ3psGVHyfAQLtuQ2R/dfkvPg== X-Received: by 2002:a05:6a21:339c:b0:1ea:ee89:5d9b with SMTP id adf61e73a8af0-1eb214de499mr32430150637.21.1737525545478; Tue, 21 Jan 2025 21:59:05 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-aab358fc6d8sm6697425a12.51.2025.01.21.21.59.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:04 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 0/7] zram: preemptible writes and occasionally preemptible reads Date: Wed, 22 Jan 2025 14:57:38 +0900 Message-ID: <20250122055831.3341175-1-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog MIME-Version: 1.0 X-Stat-Signature: r8a7i19sy6etgy4n83kjeds1fg4nzpiy X-Rspamd-Queue-Id: CF93020006 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1737525546-869934 X-HE-Meta: U2FsdGVkX1+oeEFGvH8jmYotiLZQlo/YUfHFB6Q/FSMU0sgWuFDK3DwrKl+grW1Bb+mWM4F9EHVRQTRTC3VZF8vNLCD1T65mDpCEy3OWiGarZ8gIcfQbv7eKagYngnnTQfID3g3qKCAef6anNC7nsyXmGa00niIg95G65/hD3lk2wHgRS+545sqJE16kN4tgA21cqpfPa2QSTHPV69yYbPpojuK1gK6MQYX35Gw4TvyMi+9xIWoGj8Omc5E5LBB4EnL42tf2RPntW/4TuJWKLNR0nJa2LDSuVkRZoSfJqvuDby2clG6nYb0/ueUOvmY4TGXd45Xjc7czc4Yu3eAEVuTROADjetiCHpcPdUEr+Uqd/0e0GBo7AMsN70DQx38Ded6+Gdnm6ZdVI6iGcoXTu5zw+TQO1zO97EysCFFFLgL5kuoGSjwtw/xb3VBY2R4trSnLcTWxA1s7H13qam6Gv5kGhrvD6jhpM1KEFk6rzYVa9qp+oJfP9FR2xA4CjV9HA9t1SPIiJpzqjQbPtsgA+7vcm6Hvev4TsIjP/GiiLHsgf9r0ra1mZ32hdJMgFhTEfbpidvPIwALpa/2ge1E2Q1KATOZTdMbDK4aosQDJ65/kioOtkPm+/MV1OBvgvLN/nHuIFDwJSI8wuMs6hLBOEPcrMSINd8mpGjXDxJVaA2QnpdS2SM+K3pWJsY7ZBw4i6IiJ2mxaLk/u5zxnE7olHqiDuv+tLCykHAkxWSTd3x8sThk6Us7spSS1Iug57IOdeeUKhdiLFfWhRZPt4t4QIhDeI2wIX9/4hogTOL1sPiRuRlHLUBwKuRX0jBTodLiuiwML4KrYRZinQZgIz4Vy6bCQav6GUwVEy9Mqp4wM58giJB+31sSd0MDSPyId8/md5k6b4AxfXioebev7LlN6gRO0WvE3GTizi3kWe1ifFBFxW6Z4mVNiENsDzTRKUdSPyqmMAO9zZfqCQ7Jg5ke hdN7gNBs hKJB6ZAX4fdhACf6NSmc5BFEMYYowaviBmuLDjS8rvfe7WYarsB6K70+GBB712saxLNxDSp49WpuAVd22iKWV0N1Oef0zwfv/x5R2Aqm5FoYcRZZmCMOa867reO2ujsh1iVOYGL5p7KvMCLxEeL+xunJKVwDFRuf8FHNjauSuZVzVoQM2UTUBBcwwvE35kQEEtrSeWpr2v+bk4KVA/z/2MO/cQa/T+hxtlIXKpj5Q8meTvgkRP7N+ys67ICpWmlg1wn+UwKO5EvGTTBqYK2F4e3C2dKi5uP6L5snXqKlnq1e87/X85awJa63dhvaanfrm9kDR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is Part I (aka Episode IV), which only changes zram and seems like a good start. More work needs to be done, outside of zram, in order to make reads() preemptible in a non-occasional manner. That work will be carried out independently. There are more details in the commit messages, but in short: Currently zram runs compression and decompression in non-preemptible sections, e.g. zcomp_stream_get() // grabs CPU local lock zcomp_compress() or zram_slot_lock() // grabs entry spin-lock zcomp_stream_get() // grabs CPU local lock zs_map_object() // grabs rwlock and CPU local lock zcomp_decompress() Potentially a little troublesome for a number of reasons. For instance, this makes it impossible to use async compression algorithms or/and H/W compression algorithms, which can wait for OP completion or resource availability. This also restricts what compression algorithms can do internally, for example, zstd can allocate internal state memory for C/D dictionaries: do_fsync() do_writepages() zram_bio_write() zram_write_page() // become non-preemptible zcomp_compress() zstd_compress() ZSTD_compress_usingCDict() ZSTD_compressBegin_usingCDict_internal() ZSTD_resetCCtx_usingCDict() ZSTD_resetCCtx_internal() zstd_custom_alloc() // memory allocation Not to mention that the system can be configured to maximize compression ratio at a cost of CPU/HW time (e.g. lz4hc or deflate with very high compression level) so zram can stay in non-preemptible section (even under spin-lock or/and rwlock) for an extended period of time. Aside from compression algorithms, this also restricts what zram can do. One particular example is zram_write_page() zsmalloc handle allocation, which has an optimistic allocation (disallowing direct reclaim) and a pessimistic fallback path, which then forces zram to compress the page one more time. This series changes zram to not directly impose atomicity restrictions on compression algorithms (and on itself), which makes zram write() fully preemptible; zram read(), sadly, is not always preemptible. There are still indirect atomicity restrictions imposed by zsmalloc(). Changing zsmalloc to permit preemption under zs_map_object() is a separate effort (Part II) which will require some coordination. Sergey Senozhatsky (7): zram: switch to non-atomic entry locking zram: do not use per-CPU compression streams zram: remove two-staged handle allocation zram: permit reclaim in zstd custom allocator zram: permit reclaim in recompression handle allocation zram: remove writestall zram_stats member zram: unlock slot bucket during recompression drivers/block/zram/backend_zstd.c | 11 +- drivers/block/zram/zcomp.c | 162 +++++++++-------- drivers/block/zram/zcomp.h | 17 +- drivers/block/zram/zram_drv.c | 277 ++++++++++++++++-------------- drivers/block/zram/zram_drv.h | 9 +- include/linux/cpuhotplug.h | 1 - 6 files changed, 252 insertions(+), 225 deletions(-)