[2/2] mm, rmap: handle anon_vma_fork() NULL check inline

Commit Message

Jinjie Ruan Dec. 9, 2024, 1:25 p.m. UTC

Check the anon_vma of pvma inline so we can avoid the function call
overhead if the anon_vma is NULL.

Signed-off-by: Jinjie Ruan <[e-mail]>
---
 include/linux/rmap.h | 12 +++++++++++-
 mm/rmap.c            |  6 +-----
 2 files changed, 12 insertions(+), 6 deletions(-)

Comments

Matthew Wilcox (Oracle) Dec. 9, 2024, 1:35 p.m. UTC | #1

On Mon, Dec 09, 2024 at 09:25:49PM +0800, Jinjie Ruan wrote:
> Check the anon_vma of pvma inline so we can avoid the function call
> overhead if the anon_vma is NULL.

This really gets you 1% perf improvement?  On what hardware?

Jinjie Ruan Dec. 10, 2024, 2:25 a.m. UTC | #2

On 2024/12/9 21:35, Matthew Wilcox wrote:
> On Mon, Dec 09, 2024 at 09:25:49PM +0800, Jinjie Ruan wrote:
>> Check the anon_vma of pvma inline so we can avoid the function call
>> overhead if the anon_vma is NULL.
> 
> This really gets you 1% perf improvement?  On what hardware?

Yes，the total improvement of this two patches is about 1% on our
last-generation arm64 server platform.

During the test of Unixbench single-core process creation, the trace
result shows that the two functions are frequently invoked, and a large
number of check NULL and returned.

> 
>

Patch

diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index 683a04088f3f..9ddba9b23a1c 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -154,7 +154,17 @@  void anon_vma_init(void);	/* create anon_vma_cachep */
 int  __anon_vma_prepare(struct vm_area_struct *);
 void unlink_anon_vmas(struct vm_area_struct *);
 int anon_vma_clone(struct vm_area_struct *, struct vm_area_struct *);
-int anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *);
+
+int __anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *);
+static inline int anon_vma_fork(struct vm_area_struct *vma,
+				struct vm_area_struct *pvma)
+{
+	/* Don't bother if the parent process has no anon_vma here. */
+	if (!pvma->anon_vma)
+		return 0;
+
+	return __anon_vma_fork(vma, pvma);
+}
 
 static inline int anon_vma_prepare(struct vm_area_struct *vma)
 {
diff --git a/mm/rmap.c b/mm/rmap.c
index c6c4d4ea29a7..06e9b68447c2 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -331,16 +331,12 @@  int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
  * the corresponding VMA in the parent process is attached to.
  * Returns 0 on success, non-zero on failure.
  */
-int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)
+int __anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)
 {
 	struct anon_vma_chain *avc;
 	struct anon_vma *anon_vma;
 	int error;
 
-	/* Don't bother if the parent process has no anon_vma here. */
-	if (!pvma->anon_vma)
-		return 0;
-
 	/* Drop inherited anon_vma, we'll reuse existing or allocate new. */
 	vma->anon_vma = NULL;