From patchwork Tue Feb 6 21:50:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 13547801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27328C4829A for ; Tue, 6 Feb 2024 21:50:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 165936B0078; Tue, 6 Feb 2024 16:50:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C7036B007E; Tue, 6 Feb 2024 16:50:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E15D78D0002; Tue, 6 Feb 2024 16:50:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BC7BA6B0078 for ; Tue, 6 Feb 2024 16:50:37 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8D618A058A for ; Tue, 6 Feb 2024 21:50:37 +0000 (UTC) X-FDA: 81762723714.03.3523840 Received: from out-175.mta0.migadu.com (out-175.mta0.migadu.com [91.218.175.175]) by imf02.hostedemail.com (Postfix) with ESMTP id E35EB80006 for ; Tue, 6 Feb 2024 21:50:34 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=oQ5zAB3t; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf02.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.175 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707256235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PMnu4dKfXWpdSsa89ceamn3rfdEYh+qeIqCY1GsWG3s=; b=bi5Gb1JF1NTu5uDtzkov/beJ6qKBxag496u6zeyd01q/3uX2nXaQkMLT7KU5peWuaeEl70 BoCreMtTvsee09yaKYq58rTARVY8B4GitvG+FU3dkiKHEGFr0qZPviniivrcB7VKEcdxAl AxNv5JTl8DrIDHE3w3uzdeVoyJ4kF/4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=oQ5zAB3t; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf02.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.175 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707256235; a=rsa-sha256; cv=none; b=GuJpyr1MPDuHGWmN7N7Iai/ECCuSZN0+R2j+P/0mok8XXO8S5cMM61y6obWzDNXjVUmN+n FGnDWyQFo9WjrggzFY+RmOZiRAIH3/c4nPWk38Xo5l6Sfy9/UA9TUDXzeHePm+9HHdKwc6 ZATu69nqHt961+CKoF0nUZqHGhgFnSY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1707256233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PMnu4dKfXWpdSsa89ceamn3rfdEYh+qeIqCY1GsWG3s=; b=oQ5zAB3taD+tTxyVw3ipX7rxGNX/XcZzEzLjTlWnsB44TnnNwKJrkeG9lJftcBT3ARlQrj 49akRKF7lx7OsTsEMkE/jD0vaJE0VXQlWs8xsxMbOzaUsujG9bw0jfinyefXlxPleCWtV+ 5mGygACeGMCme5Jeb/Jd3GuJmfiQWYU= From: Kent Overstreet To: linux-mm@kvack.org Cc: Kent Overstreet , Vlastimil Babka , Matthew Wilcox , Michal Hocko , "Darrick J . Wong" Subject: [PATCH 1/3] mm: introduce memalloc_flags_{save,restore} Date: Tue, 6 Feb 2024 16:50:14 -0500 Message-ID: <20240206215016.961253-2-kent.overstreet@linux.dev> In-Reply-To: <20240206215016.961253-1-kent.overstreet@linux.dev> References: <20240206215016.961253-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: E35EB80006 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ytdtgkeh6j8aza1rux3gethb1fprrw5c X-HE-Tag: 1707256234-321207 X-HE-Meta: U2FsdGVkX19rGWInyfDFeGDbzshpumC4+MN1Oglz2kg34oy9EeTv92jjQMJt8BzEI0J3G4zKjhdokYAyI5olPeieEHesxkY7+fDT8wZ7N2W/Trrx37G+oX+NofH5+c4rtREwLprRinw0tAoqjewvMxeAWyfK428udaUNyCbUBW0APiPTqfGvMxmjj9vs/q1SZZ6pZmDYCp8etrpVPRtmVvOg2cvo/KQVD/21m6SdbxK8bHIMhWOXgQJ6WuH8SKNxivSZ7mTK6lnvwp/lguu/ly83aMPABzS6zrnGWtAhyYCf0TzyRINBwatpTw8PFW+qnd/2JsVB/Mf0Jj7ufBMKgdZ4KVYeoLhR3VoxXJmib2gahwCcxq2B6xS3sux6RCDHtfbJEfdHCQ757d9DZmc6rYPtN/pX3B/v3ohqIkWVke5PAUCDceEW52EvWhwcwMytYUnN1GfCoctXT3vl6a74eyj+9nC2v7r8CIWEJoC2BP3mCRAmEEI4fAePYsB9FAss712CpYrMP5NiOvjTdj1k0EIdKPPRWncSb6nR+s6C+uQ/flfzUE6rpIKDXDf7y93w+3XEh0t8tzOf0vJPGgzI/cxWmEJzJ5BfmyMVtlIVtvgMTf86LsjwDJY0F4YNmkrbhAEVkSzOO6dgu23sfH4PC6MZkAnVE16xDAn18lxTlOOFp835KTmWrnkeJuYYRVi1o3BLEHV+d6e/qXo3OBiQvrZIqZYeDYVJRBCA7TiVQPJEfesgwUsTr1zzXzc7mdy2NZhVOGLnj/VCGoEtgn1E1G5G7q3bnHKx3MCLdQdAwsNUi7TJ1Lzhdok670o0Di98kvmd428YUJ6Cp5dwwepd5ZLO6WPm2W50RdkztMVJs4awK4Ghr6TSG15fHvgABbGC/dCcFLLD1OHeDpkGizOqQvVMLvW5lQ4D04yqWD3LUtfLGS5pl39cxXcacjlVZ0Gtnf1JGA56V8QgXqAMg8i enveeSmw A9+HAEzqjHYqMIFnG8bYKZi5pWvyr32GQaiSc+H2VXhTjqW6IDgPKc4NyEyYdPQQvhi7ScMrO7qRnb4K9SCfzeOegSyqMdlWq3SDYEecTnGet09NgM7jcLr4XnpEltv55w69MkmZWNOSwG6h92pU6/0PPVBzjJ04wSRAzWFTAkQxpTIAKnyuF8rX1UqePu375n+/brrz+znzwjRsPjdagKXvjeqIDM9ITpin0zEcWkHf866ti48I32fzurKQ8YO+FDDwpDWaG+vlE7xkzVL5Y0zDL/nw3L821/32l3yn+mAejZ+qVc78QHu2puw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Our proliferation of memalloc_*_{save,restore} APIs is getting a bit silly, this adds a generic version and converts the existing save/restore functions to wrappers. Signed-off-by: Kent Overstreet Cc: Vlastimil Babka Cc: Matthew Wilcox Cc: Michal Hocko Cc: Darrick J. Wong Cc: linux-mm@kvack.org Acked-by: Vlastimil Babka --- include/linux/sched/mm.h | 43 ++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 9a19f1b42f64..f00d7ecc2adf 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -306,6 +306,24 @@ static inline void might_alloc(gfp_t gfp_mask) might_sleep_if(gfpflags_allow_blocking(gfp_mask)); } +/** + * memalloc_flags_save - Add a PF_* flag to current->flags, save old value + * + * This allows PF_* flags to be conveniently added, irrespective of current + * value, and then the old version restored with memalloc_flags_restore(). + */ +static inline unsigned memalloc_flags_save(unsigned flags) +{ + unsigned oldflags = ~current->flags & flags; + current->flags |= flags; + return oldflags; +} + +static inline void memalloc_flags_restore(unsigned flags) +{ + current->flags &= ~flags; +} + /** * memalloc_noio_save - Marks implicit GFP_NOIO allocation scope. * @@ -319,9 +337,7 @@ static inline void might_alloc(gfp_t gfp_mask) */ static inline unsigned int memalloc_noio_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC_NOIO; - current->flags |= PF_MEMALLOC_NOIO; - return flags; + return memalloc_flags_save(PF_MEMALLOC_NOIO); } /** @@ -334,7 +350,7 @@ static inline unsigned int memalloc_noio_save(void) */ static inline void memalloc_noio_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC_NOIO) | flags; + memalloc_flags_restore(flags); } /** @@ -350,9 +366,7 @@ static inline void memalloc_noio_restore(unsigned int flags) */ static inline unsigned int memalloc_nofs_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC_NOFS; - current->flags |= PF_MEMALLOC_NOFS; - return flags; + return memalloc_flags_save(PF_MEMALLOC_NOFS); } /** @@ -365,32 +379,27 @@ static inline unsigned int memalloc_nofs_save(void) */ static inline void memalloc_nofs_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC_NOFS) | flags; + memalloc_flags_restore(flags); } static inline unsigned int memalloc_noreclaim_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC; - current->flags |= PF_MEMALLOC; - return flags; + return memalloc_flags_save(PF_MEMALLOC); } static inline void memalloc_noreclaim_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC) | flags; + memalloc_flags_restore(flags); } static inline unsigned int memalloc_pin_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC_PIN; - - current->flags |= PF_MEMALLOC_PIN; - return flags; + return memalloc_flags_save(PF_MEMALLOC_PIN); } static inline void memalloc_pin_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC_PIN) | flags; + memalloc_flags_restore(flags); } #ifdef CONFIG_MEMCG From patchwork Tue Feb 6 21:50:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 13547800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE459C4828D for ; Tue, 6 Feb 2024 21:50:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8BB76B007D; Tue, 6 Feb 2024 16:50:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B9E6E8D0001; Tue, 6 Feb 2024 16:50:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A09A6B007E; Tue, 6 Feb 2024 16:50:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 863816B0078 for ; Tue, 6 Feb 2024 16:50:37 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 59CA51C1499 for ; Tue, 6 Feb 2024 21:50:37 +0000 (UTC) X-FDA: 81762723714.01.F304EC9 Received: from out-185.mta0.migadu.com (out-185.mta0.migadu.com [91.218.175.185]) by imf28.hostedemail.com (Postfix) with ESMTP id 9371EC0010 for ; Tue, 6 Feb 2024 21:50:35 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tk4aRtnU; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf28.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.185 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707256235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+KzZqwbQeNdV0pmvYQSEwIojyt3c9wDtFh98zm28exM=; b=nIu+lApiwOpmQISN0GKXcUsJt86wMrf+yaAPE4oUd4v2m2O+w1vX7ntR+XCYuJaqf2v65v luve5dlK5URDNKVo9qQoyFYoukunoPpwxb/SlP1+JUEt4XtXmAQpJaG2CzWOwqdZBP/pX0 YF4BDj4VbcqxjjQ7d+CegLBKqbNT1RQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tk4aRtnU; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf28.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.185 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707256235; a=rsa-sha256; cv=none; b=RJAFIPy+sNE5y6sDy0p4BHAIGN09bSWHGlNRO9tUHM7qBEdftGkJeAcItxxq8jE23LkqKw YqCrSpf/LU7ESovD9gAgYKsJMZWe5Qu+V79mDhs3iaNf7QwH6hDnoHggQ35aLdA4KedCsn cf6i+9FILidz6h6FA4JXDQhLnJsRoME= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1707256234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+KzZqwbQeNdV0pmvYQSEwIojyt3c9wDtFh98zm28exM=; b=tk4aRtnUs9D7/a8Awzi/3OrDatNqSqVL1UJmReDHOs4OtWheTZ/h945zMwvR6FehTmO0uX GZFY+7X5vsUH/z9OWSb7WaMHZVHSwQL6Bpmw10ClU2wqRPHHOeoxMR6PXX99zHEkiRv+T5 hlgGPU80j9BwnG+bEiHQF9tx/esusY0= From: Kent Overstreet To: linux-mm@kvack.org Cc: Kent Overstreet , Vlastimil Babka , Matthew Wilcox , Michal Hocko , "Darrick J . Wong" Subject: [PATCH 2/3] mm: introduce PF_MEMALLOC_NORECLAIM, PF_MEMALLOC_NOWARN Date: Tue, 6 Feb 2024 16:50:15 -0500 Message-ID: <20240206215016.961253-3-kent.overstreet@linux.dev> In-Reply-To: <20240206215016.961253-1-kent.overstreet@linux.dev> References: <20240206215016.961253-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 9371EC0010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: j38u8gm5u75137dhtrj4ijfiub8rq6h4 X-HE-Tag: 1707256235-6930 X-HE-Meta: U2FsdGVkX19hpIdJtuQuYqLCOXxILwbKf9FFxwXKuNNGsHIk+lX0zK9+0AEB1JfMi59hsL3Qo1VvrK3nK8FGe4VCj68xCIEFMV5KhQGeWzM/BbYwoXmyTNKlJukXf2Jq5R9ywHQFvtUGdOos7Lmo/yy8Cq/QDIwcV5KZhyq89Nw/He0BmxK6tTEM+vb5qdPKFJ49eeUowGG+y2tAoTb+6KVHLvbyTvc/+uGQnm82PhA3R5ITgso36p4fIPBO4tOEjSvwImLmuN3pEQAe5X/TmgNTk+GG0PskvpZ+Md2mBX30brkPPHsBBS/hgFpvhTr0tg4QFKndwu7klNu94EmLmkcX4g6hrqAVDud1qSnKUSfo7mGUpoPuiPqYkCcoYpA9JTnUBVnEfgPK0Vq82uyPffRioWVXcS3xnQHi1QuoZp1I/Mo0Tj66zR4idpMEc4bKW6EyGJ2d0BrMBg/bT4/0Bpq/YQLI1nrSusDOyQHa9zvLPvAvbPok2bnWuNLVJz/9zzXo9cercWV+lXvGeKErLdpPEx2yrnvlQx9973qLsQinOLOUm5lJ32EqD4cQzvxm8wxoye1rnLE+7Y2z3TsIC2ISIpxnSXhqK4/VJAdcROy+qurNQY6hUyj7skwY40KVbWrlbX+OQsMZHmlUZiY/PfQPYIq/jedkBZPkwic2n6VmQ/1LIiWkSr7KB44r+nwNzLQUsFZOI/nRPdJcBWzwYM4D/mhQKMuA5Rj/VA1gT5dKiiSJyOPvyRFsrpEPF75c8U3sWE9WCKeCu96IDOHIf11ZiDAtDVoRNcVTk7LN76mXMuTW3Rv2Dgtl0Sn1a/V8+YbeyxQ+7/q0aJFDg9DRT+6UXmGcdiUzXPFniJwgR81ydcx14u/UzJepCkFb2t2keXtOHhjJh1X2W0tDNGI1YAbtMt+s+dAEYXb6yWb6DH5DyesVyap4I45UnYDXrkuVF/0UD1I4c3i1891Od3f K1aD6F1B VxkJ4QtnZEVjvFsFROfinyxxDr4xLp3wp8D9Ulrocuwvn/VF7vwjnh6msQ6rO8mqPzagntjiCTgme7rrSBtnsYVkOt1b/Y1+R8qMg47T5LR2ZhYoaCjimisPMSRhDoZ4IVcbaxshfYay8am2nuuJupBKkXF3AbcQFyJBVTfvaQ+8h4LGKSFvIODeqZcjyZmhAxI1Fl3o4A1hUf4Cn6/FuJXz7kCrs1RQ8xeHAoEHBZ02JIQ0v0xZCVeSX2JdiW3Df7u8pbp0wWZiSM52PW9BWYddLddiJp8lJfwFZT1hxPsbjxng= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce PF_MEMALLOC_* equivalents of some GFP_ flags: PF_MEMALLOC_NORECLAIM -> GFP_NOWAIT PF_MEMALLOC_NOWARN -> __GFP_NOWARN Cc: Vlastimil Babka Cc: Matthew Wilcox Cc: Michal Hocko Cc: Darrick J. Wong Cc: linux-mm@kvack.org Signed-off-by: Kent Overstreet --- include/linux/sched.h | 4 ++-- include/linux/sched/mm.h | 17 +++++++++++++---- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..ca08d92b20ac 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1755,8 +1755,8 @@ extern struct pid *cad_pid; * I am cleaning dirty pages from some other bdi. */ #define PF_KTHREAD 0x00200000 /* I am a kernel thread */ #define PF_RANDOMIZE 0x00400000 /* Randomize virtual address space */ -#define PF__HOLE__00800000 0x00800000 -#define PF__HOLE__01000000 0x01000000 +#define PF_MEMALLOC_NORECLAIM 0x00800000 /* All allocation requests will inherit __GFP_NOWARN */ +#define PF_MEMALLOC_NOWARN 0x01000000 /* All allocation requests will inherit __GFP_NOWARN */ #define PF__HOLE__02000000 0x02000000 #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index f00d7ecc2adf..c29059a76052 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -236,16 +236,25 @@ static inline gfp_t current_gfp_context(gfp_t flags) { unsigned int pflags = READ_ONCE(current->flags); - if (unlikely(pflags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_PIN))) { + if (unlikely(pflags & (PF_MEMALLOC_NOIO | + PF_MEMALLOC_NOFS | + PF_MEMALLOC_NORECLAIM | + PF_MEMALLOC_NOWARN | + PF_MEMALLOC_PIN))) { /* - * NOIO implies both NOIO and NOFS and it is a weaker context - * so always make sure it makes precedence + * Stronger flags before weaker flags: + * NORECLAIM implies NOIO, which in turn implies NOFS */ - if (pflags & PF_MEMALLOC_NOIO) + if (pflags & PF_MEMALLOC_NORECLAIM) + flags &= ~__GFP_DIRECT_RECLAIM; + else if (pflags & PF_MEMALLOC_NOIO) flags &= ~(__GFP_IO | __GFP_FS); else if (pflags & PF_MEMALLOC_NOFS) flags &= ~__GFP_FS; + if (pflags & PF_MEMALLOC_NOWARN) + flags |= __GFP_NOWARN; + if (pflags & PF_MEMALLOC_PIN) flags &= ~__GFP_MOVABLE; } From patchwork Tue Feb 6 21:50:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 13547802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A82C4828D for ; Tue, 6 Feb 2024 21:50:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BDC68D0002; Tue, 6 Feb 2024 16:50:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 61EC58D0001; Tue, 6 Feb 2024 16:50:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 471B28D0002; Tue, 6 Feb 2024 16:50:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 225C88D0001 for ; Tue, 6 Feb 2024 16:50:38 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D0009802CB for ; Tue, 6 Feb 2024 21:50:37 +0000 (UTC) X-FDA: 81762723714.01.DA0C536 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) by imf09.hostedemail.com (Postfix) with ESMTP id 1B8C9140020 for ; Tue, 6 Feb 2024 21:50:35 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=njyZS2S1; spf=pass (imf09.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707256236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lhFR/wbOdeCSFwsNHKkPjNcdJPS60NDD1V/4AxNlPWM=; b=ef7YQ01KKiRlK+7PNnFQkF9PBeqdzZK4wc+SQcUisMi1PPfNNNd7lqGb+YEkesbm8AdBGV YBx/Zj79RvUN6M3qJOXvvEj0Wk+hsmV5Nx8dPMKgHvBCgwmM5Wv47c0TL8MKFQfs9Yot4q ua/GQEY8u+rI7Oxz+1VFRWk9/uim8rI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=njyZS2S1; spf=pass (imf09.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707256236; a=rsa-sha256; cv=none; b=OebJHR6TDgpDc2CuaUvxXjQwTAktrWif+H0tf42WjZX5csz9Um4mUCiNhjyN1BcXn0zZ94 SwkUq85lO/s5ochFJUWq2Q40CrZeth64yH2bvaZTQIvqloZJ02qKBfw5KX0BT6TBc/YCqZ aFucBuKdIVdw5UZN9prmal4406Chanc= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1707256234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lhFR/wbOdeCSFwsNHKkPjNcdJPS60NDD1V/4AxNlPWM=; b=njyZS2S1hI5M7S33TRsxwRlJbSxbJufYZnbtKSdgdF22dU1WQVz37csOfkSfvWpqOpbvbM hPKT2AfkvTB4dABGyMplO6l6rli+BIGFqYki85vQVcaN7CAhAvhr5tN/7R558wsMIDfgN7 kKeagKkyNAWs922DQ4r63oHjHhwZGhU= From: Kent Overstreet To: linux-mm@kvack.org Cc: Kent Overstreet Subject: [PATCH 3/3] mempool: kvmalloc pool Date: Tue, 6 Feb 2024 16:50:16 -0500 Message-ID: <20240206215016.961253-4-kent.overstreet@linux.dev> In-Reply-To: <20240206215016.961253-1-kent.overstreet@linux.dev> References: <20240206215016.961253-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 1B8C9140020 X-Rspam-User: X-Stat-Signature: micykq1znqynzkr8bdjsu5abi3ekkshj X-Rspamd-Server: rspam01 X-HE-Tag: 1707256235-545656 X-HE-Meta: U2FsdGVkX1+6odNQCx34dQ4jLjq3JtoL4k2253Tb14GlqFD9ZiOD2fZFeBjiITWc3yLoJsigofO78S36ae2jjT9B6R2D+iIvJCvieaz7RyQjjxbyH+kCV8YkvQzcoTugaQqNzxjVDGcVZqP6DsVc2UOnSedZHao58NWC+ffBdpz1MQAKm/DhzFFXUgVLcD84fgYKJjCmFjuqRRZzAFXG0i1EiJBNeKaM21QqrbcGThJf2YDFbI551bkyg77U4dfyqsqwTwS73FqV1HTPH3Mbx95yJj9/mGOGJ6ByEegh9gdI3Aw9vX3V00ouVCaJOWF7oO3fHJTVzmS84ndoEQMNqnDf6+exqMkK8PMCsX1EmNeKwvHxeXQFVyL/5CfZtqEWz/0TtzeCdQoKpQaZ1e9j+CKM6oNYbKm2jDM0pDIm2E6lJagQqLYNQB9tb3J/DRkIPnp7Uv/xsD0stgZLNwVdvVoc0QjpjZXbAblW6XzZufMjqxjorqagwiL09uEItK4/gvIW/Ph02cut6JpYP5lHqL782OBp5ApL3k9eSy0+vli7NTPGrr9I5Z9+DkKUmB7EcvSzj+nPz0Lnpz/tfsLBd9M9stX+gksA7W97teiwLLvgtm02W3U61hybHLFjt7M6DBAKIEDKf65Ql5NqwBxgbImBUeqd8KiQliEGlpGIbmE5Bmu8GcIqovDhSepJweTYenncjEvxWa1yxr7dHYKtwjvPy2Tiu5kYCXfUQsSvAJWn4XUlMQGfTxFmP9OQCiRfETGQMOaa+Y0TOADn1oKTZuOJwE17WlP+Uu1unxlnpRuzxaefw/PDvRRMBItloFN6gWC+jb8QsPWa35pLeGMRQhRCwfnfOn7toIvqzhVl4rFdmPVuytGgvM31jfX9j4uOs/hqq13Xq7IGzzWT22O4QdGUmCqh3+uDlaeqiIPYh9597YNW1+R0AeUZBCXoTt02eFIdlFfPrLXRaDpQVrG trHXjUy1 eXzoP7ZQzxWIU6ZkjVvOBZzr3sSd8rp/PNjgTd0Bbrc2jBY2T92ltfM2AFJYzhZ3Ejsn16/vh7Yd0XUeZzB3ykZ3f9Qm+vlUU+pvQC1MREcvWPKkMeko1G5E+9CYIGpE2Cbz6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add mempool_init_kvmalloc_pool() and mempool_create_kvmalloc_pool(), which wrap kvmalloc() instead of kmalloc() - kmalloc() with a vmalloc() fallback. This is part of a bcachefs cleanup - dropping an internal kvpmalloc() helper (which predates kvmalloc()) along with mempool helpers; this replaces the bcachefs-private kvpmalloc_pool. Signed-off-by: Kent Overstreet Cc: linux-mm@kvack.org --- include/linux/mempool.h | 13 +++++++++++++ mm/mempool.c | 13 +++++++++++++ 2 files changed, 26 insertions(+) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 4aae6c06c5f2..92181401ea96 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -94,6 +94,19 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) (void *) size); } +void *mempool_kvmalloc(gfp_t gfp_mask, void *pool_data); +void mempool_kvfree(void *element, void *pool_data); + +static inline int mempool_init_kvmalloc_pool(mempool_t *pool, int min_nr, size_t size) +{ + return mempool_init(pool, min_nr, mempool_kvmalloc, mempool_kvfree, (void *) size); +} + +static inline mempool_t *mempool_create_kvmalloc_pool(int min_nr, size_t size) +{ + return mempool_create(min_nr, mempool_kvmalloc, mempool_kvfree, (void *) size); +} + /* * A mempool_alloc_t and mempool_free_t for a simple page allocator that * allocates pages of the order specified by pool_data diff --git a/mm/mempool.c b/mm/mempool.c index 734bcf5afbb7..343f0ef6b69c 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -543,6 +543,19 @@ void mempool_kfree(void *element, void *pool_data) } EXPORT_SYMBOL(mempool_kfree); +void *mempool_kvmalloc(gfp_t gfp_mask, void *pool_data) +{ + size_t size = (size_t)pool_data; + return kvmalloc(size, gfp_mask); +} +EXPORT_SYMBOL(mempool_kvmalloc); + +void mempool_kvfree(void *element, void *pool_data) +{ + kvfree(element); +} +EXPORT_SYMBOL(mempool_kvfree); + /* * A simple mempool-backed page allocator that allocates pages * of the order specified by pool_data.