From patchwork Thu Oct 24 20:52:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13849704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93B49D10374 for ; Thu, 24 Oct 2024 20:52:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 256796B0088; Thu, 24 Oct 2024 16:52:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DEE56B008A; Thu, 24 Oct 2024 16:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 030E46B008C; Thu, 24 Oct 2024 16:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DCC5C6B0088 for ; Thu, 24 Oct 2024 16:52:39 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5D176C15AD for ; Thu, 24 Oct 2024 20:52:19 +0000 (UTC) X-FDA: 82709693178.20.D7CA2FD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf03.hostedemail.com (Postfix) with ESMTP id E9F9920009 for ; Thu, 24 Oct 2024 20:52:28 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4c7gyeyp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3lLMaZwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3lLMaZwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729803118; a=rsa-sha256; cv=none; b=uC8m2BNtR0BMAZrOp4GwlsL/HUor5w+V2xNtVcLMkMvW0BIuzU+794NDZVFElsACa5a1WE R3w2W0SHj0AHk9eouo4lnezhx6Rt1n6lA1RwtvIe5yYN8CMeyWHDw4rv6THI0+OGBSZnQg 4E+txBHZVlnJJ8vE/evJDqzjZA69Fms= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4c7gyeyp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3lLMaZwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3lLMaZwYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729803118; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D0cyo47lZvYDA3B5yj4Am2lDN/6HgoSPrlBQwJwKaSA=; b=uKcdMIaBwIEVN1nXmmzDSBWG6DDbAs2KpSTjeVFL/W8LeC9YgZgXUazkCTJIsMq21adWRC qrSdC5LAUClOrybuByi0PMuMv0ohb+DVBx1N97qtFfvh+KeB21mF0NQSLuKghqxRUKzlTn 32qG+aQj5Q+ckH2tRti/0EF66pZC7qA= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e376aa4586so25724707b3.1 for ; Thu, 24 Oct 2024 13:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729803157; x=1730407957; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=D0cyo47lZvYDA3B5yj4Am2lDN/6HgoSPrlBQwJwKaSA=; b=4c7gyeyp8OMkBEN7k61RDBE9fPd3HR6w2yYn+MzdY7+zV3q8oq7iJ7rAPXFHETsHtv ci7YDHouUHeW+4GqMzaslXcK4elBy1vDdL9yYA/Au7M3SCky3JZGGBy662Pz52HvXGob Sq1ScWIiNBVSHwIEX9OwvwIzdRS26F5XzXv910GYk+0X+00boKPuXlz78gb+O3Bmwoj6 N+9F8aiDZPZr6Ws2D7reX0+B07mZA6SWNd4w/pim6WhcNs/3SfHeokzwny+GYjVa3uRP 0mQ4UhKtek3IMulAPrzpDUIFkFPKfMEw8Y/QgQZS0nAabvCNaAdQBVsyZi3cGyWpr5mK OgCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729803157; x=1730407957; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D0cyo47lZvYDA3B5yj4Am2lDN/6HgoSPrlBQwJwKaSA=; b=xQ4GexXD1Y4tVPHA+CWeElxf+cAZ6uu98hWa780DnjroJOXjKXwOQ8PjA7KDCssbxp x14k8g3LMCXK0o1nCvJdwvAu2SK0o5f05+JrnTPUIni/0l7yq1xZEL0PlWSlrDgxnBF4 wQUKtAJS4HdiJmnd+YtagEKEKGuv6kaUNiLsbMFM99dw+LgbA4TGz3gMVNh/c5NmXTFF Nb6WULrx0Xv6pzgID9IJapiO5d7KzeGJp2SzkEhOC5XXOTrqfKIwyn5cfUuJ5Nbo/qaQ FzZkj3eE7Z67AOhMBRuB4DvKbEGzqxbrLuom6s8ZQTPJtKL/R6RYt3PPWWMAI+c8fn3C mQ4g== X-Forwarded-Encrypted: i=1; AJvYcCWFpLpZk8eRkKOopSuompWAkU4JmUKWzWWiqPNpk+zNWiJIbumz19CMYdZFkChR5T67tT/UsMgahw==@kvack.org X-Gm-Message-State: AOJu0YxTPkKpFnNoKFk6KaPvIWBJ0NZNwFF+2eVuUwUDTsUlJugfCdZ/ /gYuz3dxZG1ljvN9UfTJ+szOWOja4hogEx5YzGATaJai0EpVsv9+Yzv5BMrNkAYXsIX8192H6Py UmA== X-Google-Smtp-Source: AGHT+IFJIBEobTf0M/avBl1t+bIbsxZRvjcy3r9B86GReN1XHKNIk7A8yUrOFNSe7IKPSSD3SbHOyv9e5X4= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:47e4:54e:bc8e:cb18]) (user=surenb job=sendgmr) by 2002:a05:690c:7201:b0:648:fc8a:cd23 with SMTP id 00721157ae682-6e85815bef1mr1181987b3.2.1729803156923; Thu, 24 Oct 2024 13:52:36 -0700 (PDT) Date: Thu, 24 Oct 2024 13:52:31 -0700 In-Reply-To: <20241024205231.1944747-1-surenb@google.com> Mime-Version: 1.0 References: <20241024205231.1944747-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241024205231.1944747-2-surenb@google.com> Subject: [PATCH 2/2] mm: introduce mmap_lock_speculation_{begin|end} From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, andrii@kernel.org, jannh@google.com, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, mhocko@kernel.org, shakeel.butt@linux.dev, hannes@cmpxchg.org, david@redhat.com, willy@infradead.org, brauner@kernel.org, oleg@redhat.com, arnd@arndb.de, richard.weiyang@gmail.com, zhangpeng.00@bytedance.com, linmiaohe@huawei.com, viro@zeniv.linux.org.uk, hca@linux.ibm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, surenb@google.com X-Rspam-User: X-Stat-Signature: i6hha98sgmzgnbx9rwgxby453iwj9zwb X-Rspamd-Queue-Id: E9F9920009 X-Rspamd-Server: rspam02 X-HE-Tag: 1729803148-919979 X-HE-Meta: U2FsdGVkX19mUCBH+VZd2pG3qLK8Nms2EUxTBd2MR+McwM8mlsDK2R5qZ0he0HpPBsDSlRg6zK0j4bG06DMY4lMMkrvlN2ApK3+Zx83FlltUmqwjJNRksn0rjb+uik2TUmuxuhfBtB4TQdIYSSgae1E5iTMMTA/qFyr97VnFu0U7ZtFgPMVbdBP1ofQYfml7dtg5ZLD43J/F6e31cAmQAlMD4PanCFhd39ppCgl1gZuRb1sW7jntSMwJqOCJ23MPSitArCa7sVUG4MG63ijSGm66AmN3I1P5+8kVJJSk/WXg4NaGmGGqWgBg//d3EXr0LV7J1sCHV8dsGdue/ku+Sv+HNlMvaCecCw561OJA1a6EOAF5PZoibdXSj1q8zFeazT7T9LBQ+J2z525TxXt0+Yshbij0MmQ69lPzy94cWsP6C4pg7Q7aSPrFOBcrmrANThpg3lkBXG5PM9wf2pKSCi2Ftx8pxINybEJWta/rpkHC7HMNzfJIxgnfx15wXdZFAezIBtPWSETh+EpVeI3QT1S6Nqn9Ref0PLsQq2e5j2oxy9E23OEty7xYI+tGIVvpFsiXG0YsguOkIZX+x/WDzr0xabczpL5JFZd8WJI0ZwpAwx/9Y2qpcG1QBZ4OOYmJdKb1i92DWA/3dtQ6e+6huXn0DV1uByVi+bBKE2iC+URqq6xI9cLty/J6vLcX0XKy/aMQ30PFvJUbZZiO9qWd7PT3B41bS/Pk+1ZtONz/YBZMyTkP9Pqaxq2wiw6rlMQURUpzGNyGrI5fL7p7CoD0H0uXkSdoIABCKh/nJ3bAM55MGHiov+jEXc1iiV7lMgtGAeQMFcQ7/x/28TCJHe785hUCoHYBBMrcI+T1c6cTAXrLsHmRN/e8AxCXwqeuLBlcTNbmlwedCgdJlO9KFjN2gul3W3wCGHxU1UhdxkQx3AroNO2j9+JEa3P6tK2AaEVdAfkTIpsAtkhW/ftx4pj Bm1mMCJ4 UNAkzlEhmXCQXMGDABOyfFvLChFHOdEfZ41zA925R9hDBSaIEHhg/SHF578dr8Kh5nKROJ3/noBJW0QdAO/EIFKSUOTtj3Seqa0fNGNwJw41Cfk/kgkWT96f90VhpGnJFRCaJRsIWAU6VDWMGcVHeeCJg6r9sU4+riFSeVjdnQm2xtwi/LMJDemxJklN3wRwCJw7jRGMMuxHu+inIMIE0sXERDBRQar6U1annJglab/gKaW8xm6pmRrkrUiNt3i/EQaTqWs0QFdspIQZeszki0Z5fkMHatvAOuBAViux1o+ZvrrucHIq6kOSOKUrNNYTlM6E778bLRVhgj0JGjZ9AN90PsoLAsmxn9i6aV0Ez6IpcpPOl2c9UGQhndh9vV6URO1labhdSZ6VXpuOhIu/N9BejE+tavj7zUBv/9cXIyIMxURDvaequDTI/DA8Dq1eAX27Ua2qMKUJ5m13LB0Eo4um2YhITdBbBSFCljQQDeJ415yxsbEYjUYfaajt5OG2O0UYRgfqwf/+Fws351pAU5tZwEkvzoFrcCmVgc0sZBiMY9TzSq3Fn4znJvb65rb6i7hiAuOnrMZMAnmA8IBpaKhx7pL6UQhxNegwbKSfNHX85GppdrOAuYQ8m7s63kvnoJ78cnVuQG9QnH+5KVQtfAaN4gsWUqFKayArrHg3Hjlu6aBSJ810d1n+oJxlOwb22aMBMGIMsDAZp/uBK+Mt+xRsfCQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add helper functions to speculatively perform operations without read-locking mmap_lock, expecting that mmap_lock will not be write-locked and mm is not modified from under us. Suggested-by: Peter Zijlstra Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 6b3272686860..58dde2e35f7e 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -71,6 +71,7 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) } #ifdef CONFIG_PER_VMA_LOCK + static inline void mm_lock_seqcount_init(struct mm_struct *mm) { seqcount_init(&mm->mm_lock_seq); @@ -86,11 +87,35 @@ static inline void mm_lock_seqcount_end(struct mm_struct *mm) do_raw_write_seqcount_end(&mm->mm_lock_seq); } -#else +static inline bool mmap_lock_speculation_begin(struct mm_struct *mm, unsigned int *seq) +{ + *seq = raw_read_seqcount(&mm->mm_lock_seq); + /* Allow speculation if mmap_lock is not write-locked */ + return (*seq & 1) == 0; +} + +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, unsigned int seq) +{ + return !do_read_seqcount_retry(&mm->mm_lock_seq, seq); +} + +#else /* CONFIG_PER_VMA_LOCK */ + static inline void mm_lock_seqcount_init(struct mm_struct *mm) {} static inline void mm_lock_seqcount_begin(struct mm_struct *mm) {} static inline void mm_lock_seqcount_end(struct mm_struct *mm) {} -#endif + +static inline bool mmap_lock_speculation_begin(struct mm_struct *mm, unsigned int *seq) +{ + return false; +} + +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, unsigned int seq) +{ + return false; +} + +#endif /* CONFIG_PER_VMA_LOCK */ static inline void mmap_init_lock(struct mm_struct *mm) {