From patchwork Tue Oct 11 02:21:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: xu xin X-Patchwork-Id: 13003559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC1EAC433F5 for ; Tue, 11 Oct 2022 02:22:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 608468E0001; Mon, 10 Oct 2022 22:22:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 568E56B0073; Mon, 10 Oct 2022 22:22:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3941F8E0001; Mon, 10 Oct 2022 22:22:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 20F536B0072 for ; Mon, 10 Oct 2022 22:22:05 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EAD6A805A4 for ; Tue, 11 Oct 2022 02:22:04 +0000 (UTC) X-FDA: 80007068568.10.68FE865 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by imf23.hostedemail.com (Postfix) with ESMTP id 6A3AB14002B for ; Tue, 11 Oct 2022 02:22:04 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id c20so2345295plc.5 for ; Mon, 10 Oct 2022 19:22:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jsbh45Fv58HEehSJMVvF1reAzOnYH1CPjnbjNiOuTbI=; b=lgOVYuFLApBB99lxB0wJ+xs6iHGx1lEt2NfNrp2YK62xBG7cGjYeVl9kKELJ/qMHMn Zyd1vrv+Pck4AGaEvT974cU36a/3WWkDBHnuCTXvw1bc04EmlKd0Em5eipe+qGAEyUaU LF2Fn5nE/nb7YoHhIKt0V+cWuj0RHTf4QLYA0QG0VPhZnW8IKCAS8UCah3FqahzOxcG7 Ul5miU01nXsNlaIoyvq9ItU767ag1L+LNKPvPXixm6iPHOOSv5fl7pBcXW3nZUysG+IZ v5tV7aCmWU34KlM0txj6ud6e/MUyrd0QHdt4fi49yFsR/gC0GEzRYKXyVR47/GJsAc+p cLEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jsbh45Fv58HEehSJMVvF1reAzOnYH1CPjnbjNiOuTbI=; b=0HgeelnKhZmktAFC2hWurjN993EgBSlB7Fit4XU5rkX2eggCzcY2Rr8DG6sO/+y8ml XJ+rvdZM2XV0oIWN1+dGjaLNxujtlCRQZ/eAAyo9o370/thfFW8ehXmMFdPrAJsCvdQU L7EhOt48oaF8K/OeKFTckORA0O4oJHnGI4CyO53eUTQCyHpRkEbExKiVTGcnGcMbtMqR vIs+FGHkPEg5Nil7bzvmGgh30bI4OxB8cNCJdPZCsialYZchRalUJupfcU2dyY3ft8GT 4t3KWrRyi76YnxD3nPlmpOHRTpFXcFr7dPU1poIdFJe3vZwj0GFhv0e26mgtj/GPqE2S 3u2g== X-Gm-Message-State: ACrzQf208uHOnG4QkymuGwVDZtuEtu07k5aiJuuMlW55WV3ARUYir6fo 1B4vtG0rr9CklbqARbr7hpA= X-Google-Smtp-Source: AMsMyM7JW36aWaS0bLlTDqSb2ZhzhL4vqsQD6p3KYfBeOqc7tFwXhZjgSDiBMKko8MCBq+bCt5qKKA== X-Received: by 2002:a17:902:f0d4:b0:17f:72bd:74f3 with SMTP id v20-20020a170902f0d400b0017f72bd74f3mr22495074pla.109.1665454923480; Mon, 10 Oct 2022 19:22:03 -0700 (PDT) Received: from localhost.localdomain ([193.203.214.57]) by smtp.gmail.com with ESMTPSA id l24-20020a17090aaa9800b0020d51aefb82sm94655pjq.19.2022.10.10.19.22.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 19:22:03 -0700 (PDT) From: xu.xin.sc@gmail.com X-Google-Original-From: xu.xin16@zte.com.cn To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, xu xin , Claudio Imbrenda , David Hildenbrand , Xuexin Jiang , Xiaokai Ran , Yang Yang Subject: [PATCH v3 1/5] ksm: abstract the function try_to_get_old_rmap_item Date: Tue, 11 Oct 2022 02:21:50 +0000 Message-Id: <20221011022150.322212-1-xu.xin16@zte.com.cn> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221011022006.322158-1-xu.xin16@zte.com.cn> References: <20221011022006.322158-1-xu.xin16@zte.com.cn> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665454924; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jsbh45Fv58HEehSJMVvF1reAzOnYH1CPjnbjNiOuTbI=; b=0uVTS1/TNlm20bGeYRtMlXWD9rNQavmMoenh3zGLBGRzPuheFI4E4/wvmX1VnAVFKIMZ3R 1PPKqAK0ACf3bhKJ254ufG1nfgETojwodF4LGp9yk3GGMWKnhjrJwilqb7lbtzUa5bZiZu CVM2lER3EoXXkojAASjppuqdLQq1z1g= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lgOVYuFL; spf=pass (imf23.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.214.196 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665454924; a=rsa-sha256; cv=none; b=Si39g2oV4JQTw/qjlkvJDHoP7X/kda7M5vQY7JAkxuVKN+GtwPOAu1R272xolQacrR7gcn C2Qz/VsiCXv0NwO0PH30Oan1h9dIVzs4ZBOOSawGafG4yNd15fzltrCLESHO3/gZuwcRnU DRwD2k+xxWAAoOfRB+tvi6PLlsCpxjo= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lgOVYuFL; spf=pass (imf23.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.214.196 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: u3318pmb69oacg84cid8wp9zmfupedcb X-Rspamd-Queue-Id: 6A3AB14002B X-HE-Tag: 1665454924-632331 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: xu xin A new function try_to_get_old_rmap_item is abstracted from get_next_rmap_item. This function will be reused by the subsequent patches about counting ksm_zero_pages. The patch improves the readability and reusability of KSM code. Signed-off-by: xu xin Cc: Claudio Imbrenda Cc: David Hildenbrand Cc: Xuexin Jiang Cc: Xiaokai Ran Cc: Yang Yang --- mm/ksm.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index c19fcca9bc03..13c60f1071d8 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2187,23 +2187,36 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite } } -static struct ksm_rmap_item *get_next_rmap_item(struct ksm_mm_slot *mm_slot, - struct ksm_rmap_item **rmap_list, - unsigned long addr) +static struct ksm_rmap_item *try_to_get_old_rmap_item(unsigned long addr, + struct ksm_rmap_item **rmap_list) { - struct ksm_rmap_item *rmap_item; - while (*rmap_list) { - rmap_item = *rmap_list; + struct ksm_rmap_item *rmap_item = *rmap_list; if ((rmap_item->address & PAGE_MASK) == addr) return rmap_item; if (rmap_item->address > addr) break; *rmap_list = rmap_item->rmap_list; + /* Running here indicates it's vma has been UNMERGEABLE */ remove_rmap_item_from_tree(rmap_item); free_rmap_item(rmap_item); } + return NULL; +} + +static struct ksm_rmap_item *get_next_rmap_item(struct ksm_mm_slot *mm_slot, + struct ksm_rmap_item **rmap_list, + unsigned long addr) +{ + struct ksm_rmap_item *rmap_item; + + /* lookup if we have a old rmap_item matching the addr*/ + rmap_item = try_to_get_old_rmap_item(addr, rmap_list); + if (rmap_item) + return rmap_item; + + /* Need to allocate a new rmap_item */ rmap_item = alloc_rmap_item(); if (rmap_item) { /* It has already been zeroed */ From patchwork Tue Oct 11 02:22:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: xu xin X-Patchwork-Id: 13003560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B11E5C433FE for ; Tue, 11 Oct 2022 02:22:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3476C6B0072; Mon, 10 Oct 2022 22:22:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D086900002; Mon, 10 Oct 2022 22:22:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 123208E0001; Mon, 10 Oct 2022 22:22:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EAD476B0072 for ; Mon, 10 Oct 2022 22:22:22 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AEA121401E6 for ; Tue, 11 Oct 2022 02:22:22 +0000 (UTC) X-FDA: 80007069324.17.8EB6B3E Received: from mail-pj1-f67.google.com (mail-pj1-f67.google.com [209.85.216.67]) by imf02.hostedemail.com (Postfix) with ESMTP id 5173980025 for ; Tue, 11 Oct 2022 02:22:22 +0000 (UTC) Received: by mail-pj1-f67.google.com with SMTP id 70so11256355pjo.4 for ; Mon, 10 Oct 2022 19:22:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vzxRXZh2ZpCVU2bpLGiMFKwYjl0iDSX4se2xXGC6hk8=; b=g7/Dln2aP/rzIlJVDCOcKO2ExXMwc8oou+gp2lMkgFtPAxQbvBbSN+cy93oOMBZb2/ lXHlCMKPHw/Q5eRk8hfkejD4HPR0jeernWPy60zNRCTE9f+AGO/IgXblNhFliN5OjkLj G6z8xHdrdVmTdaiJ9TTn13VZvpLsVDoPymx8pukFFllSW41BqsrckdeW4cFyn7ey+JU3 tB5tDXQRLUiZiXFuVHcNVwFLOn6wHm928qtqHTugVb8BFWby7Td7nHO0d/2/4DYD9FZ0 IpQHrV4b+AQ8d5d8Tu5g3lYVM2M86t34Qfow2FFlQv42bteYGNhttdm5IlyAkY7aZsTe 3vTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vzxRXZh2ZpCVU2bpLGiMFKwYjl0iDSX4se2xXGC6hk8=; b=J3zwjziU7ipi3IFq/W3FGWdKruK5FAw9iEnS7R2PCA8tkQ/RCyMH/TJoHZqNcD9sgY PsVH9IMq8L8v4wQCMSGBjDSx0fByOfAgbE+KLlQvtFFdev5c7EUtVpn4HE97FDCIrWbc injon17G6n3nIB8Lv+10+G3mAUY+2bXB47i73XWUU2TALLLSR25Ln1zv6lASNDCnW5qZ 7S+RT4BtyfwScpTKe24IdxITz2BKd7pkXxGR+qmvG0A5QL7weoZ6rsoGUwaP1kWqdbbr RO/C0a5zeNArPc3IQ3/VKX29921nnVM6+nOob5/KEP1aQa8iOrhj9YT5SAKlwDJqxnHm gjJg== X-Gm-Message-State: ACrzQf2+S+9kA/uoHuBrUJXh5m+b8uR04hvgu0SJUSJWrak4zeuaAhDf TEe12kGtVmr2qVCs7c/TffI= X-Google-Smtp-Source: AMsMyM5V2AUbwC+A+kA+F0EbjGWUOHq5622wQRFWFgaeIPP22fPizocpRdqCp0xJv97gVKZ073kb+A== X-Received: by 2002:a17:902:c40f:b0:17f:6737:9527 with SMTP id k15-20020a170902c40f00b0017f67379527mr21820578plk.19.1665454941170; Mon, 10 Oct 2022 19:22:21 -0700 (PDT) Received: from localhost.localdomain ([193.203.214.57]) by smtp.gmail.com with ESMTPSA id n6-20020a17090a2bc600b0020a8e908dc8sm9363286pje.4.2022.10.10.19.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 19:22:20 -0700 (PDT) From: xu.xin.sc@gmail.com X-Google-Original-From: xu.xin16@zte.com.cn To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, xu xin , David Hildenbrand , Claudio Imbrenda , Xuexin Jiang , Xiaokai Ran , Yang Yang Subject: [PATCH v3 2/5] ksm: support unsharing zero pages placed by KSM Date: Tue, 11 Oct 2022 02:22:15 +0000 Message-Id: <20221011022215.322269-1-xu.xin16@zte.com.cn> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221011022006.322158-1-xu.xin16@zte.com.cn> References: <20221011022006.322158-1-xu.xin16@zte.com.cn> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665454942; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vzxRXZh2ZpCVU2bpLGiMFKwYjl0iDSX4se2xXGC6hk8=; b=LL/Uj0sKptHlEJZufadBv+k6D+M1de0lvNP1QbYEk5sJIV5cU/+LDDTNowNJbbWdcgkW7C 4YdoT+qGt4vl8SOIoS+ywL7qg61vT9L0/kYiVfkc1/22fa3fdEV7t0DsxemLvhUowlwDm+ cbNGv00x409I0rIHeNnpfWvK8oOvlqo= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="g7/Dln2a"; spf=pass (imf02.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.216.67 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665454942; a=rsa-sha256; cv=none; b=EBIr6dgOHv+uVziXTAeIdCkWgl4oQOobBjfjZWblSwSrpvw+SZJduAj9yJX1u/fMNIi2R3 evbxbwGJZ1v0I0Msdk+1IrkHa4gBZ8l8qzp2sNUFojyk7BhsInudSIokFUPYK1mT4W0G9x uKItwVNwe/5e7KXxtmXmZZXBDXCZTWk= Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="g7/Dln2a"; spf=pass (imf02.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.216.67 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: nrty4156pfbbjafyydgxwact1g56tej6 X-Rspamd-Queue-Id: 5173980025 X-Rspamd-Server: rspam01 X-HE-Tag: 1665454942-303232 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: xu xin use_zero_pages may be very useful, not just because of cache colouring as described in doc, but also because use_zero_pages can accelerate merging empty pages when there are plenty of empty pages (full of zeros) as the time of page-by-page comparisons (unstable_tree_search_insert) is saved. But when enabling use_zero_pages, madvise(addr, len, MADV_UNMERGEABLE) and other ways (like write 2 to /sys/kernel/mm/ksm/run) to trigger unsharing will *not* unshare the shared zeropage as placed by KSM (which may be against the MADV_UNMERGEABLE documentation at least). To not blindly unshare all shared zero_pages in applicable VMAs, the patch introduces a dedicated flag ZERO_PAGE_FLAG to mark the rmap_items of those shared zero_pages. and guarantee that these rmap_items will be not freed during the time of zero_pages not being writing, so we can only unshare the *KSM-placed* zero_pages. The patch will not degrade the performance of use_zero_pages as it doesn't change the way of merging empty pages in use_zero_pages's feature. Fixes: e86c59b1b12d ("mm/ksm: improve deduplication of zero pages with colouring") Reported-by: David Hildenbrand Cc: Claudio Imbrenda Cc: Xuexin Jiang Signed-off-by: xu xin Co-developed-by: Xiaokai Ran Signed-off-by: Xiaokai Ran Co-developed-by: Yang Yang Signed-off-by: Yang Yang Signed-off-by: xu xin --- mm/ksm.c | 136 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 105 insertions(+), 31 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 13c60f1071d8..e351d7b6d15e 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -213,6 +213,7 @@ struct ksm_rmap_item { #define SEQNR_MASK 0x0ff /* low bits of unstable tree seqnr */ #define UNSTABLE_FLAG 0x100 /* is a node of the unstable tree */ #define STABLE_FLAG 0x200 /* is listed from the stable tree */ +#define ZERO_PAGE_FLAG 0x400 /* is zero page placed by KSM */ /* The stable and unstable tree heads */ static struct rb_root one_stable_tree[1] = { RB_ROOT }; @@ -381,14 +382,6 @@ static inline struct ksm_rmap_item *alloc_rmap_item(void) return rmap_item; } -static inline void free_rmap_item(struct ksm_rmap_item *rmap_item) -{ - ksm_rmap_items--; - rmap_item->mm->ksm_rmap_items--; - rmap_item->mm = NULL; /* debug safety */ - kmem_cache_free(rmap_item_cache, rmap_item); -} - static inline struct ksm_stable_node *alloc_stable_node(void) { /* @@ -420,7 +413,8 @@ static inline bool ksm_test_exit(struct mm_struct *mm) } /* - * We use break_ksm to break COW on a ksm page: it's a stripped down + * We use break_ksm to break COW on a ksm page or KSM-placed zero page (only + * happen when enabling use_zero_pages): it's a stripped down * * if (get_user_pages(addr, 1, FOLL_WRITE, &page, NULL) == 1) * put_page(page); @@ -434,7 +428,8 @@ static inline bool ksm_test_exit(struct mm_struct *mm) * of the process that owns 'vma'. We also do not want to enforce * protection keys here anyway. */ -static int break_ksm(struct vm_area_struct *vma, unsigned long addr) +static int break_ksm(struct vm_area_struct *vma, unsigned long addr, + bool ksm_check_bypass) { struct page *page; vm_fault_t ret = 0; @@ -449,6 +444,16 @@ static int break_ksm(struct vm_area_struct *vma, unsigned long addr) ret = handle_mm_fault(vma, addr, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + else if (ksm_check_bypass && is_zero_pfn(page_to_pfn(page))) { + /* + * Although it's not ksm page, it's zero page as placed by + * KSM use_zero_page, so we should unshare it when + * ksm_check_bypass is true. + */ + ret = handle_mm_fault(vma, addr, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, + NULL); + } else ret = VM_FAULT_WRITE; put_page(page); @@ -496,6 +501,11 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm, return vma; } +/* + * Note: Don't call break_cow() in the context protected by + * mmap_read_lock(), which may cause dead lock because inside + * break_cow mmap_read_lock exists. + */ static void break_cow(struct ksm_rmap_item *rmap_item) { struct mm_struct *mm = rmap_item->mm; @@ -511,10 +521,35 @@ static void break_cow(struct ksm_rmap_item *rmap_item) mmap_read_lock(mm); vma = find_mergeable_vma(mm, addr); if (vma) - break_ksm(vma, addr); + break_ksm(vma, addr, false); mmap_read_unlock(mm); } +/* Only called when rmap_item->address is with ZERO_PAGE_FLAG */ +static inline int unshare_zero_pages(struct ksm_rmap_item *rmap_item) +{ + struct mm_struct *mm = rmap_item->mm; + struct vm_area_struct *vma; + unsigned long addr = rmap_item->address; + int err = -EFAULT; + + vma = vma_lookup(mm, addr); + if (vma) + err = break_ksm(vma, addr, true); + + return err; +} + +static inline void free_rmap_item(struct ksm_rmap_item *rmap_item) +{ + if (rmap_item->address & ZERO_PAGE_FLAG) + unshare_zero_pages(rmap_item); + ksm_rmap_items--; + rmap_item->mm->ksm_rmap_items--; + rmap_item->mm = NULL; /* debug safety */ + kmem_cache_free(rmap_item_cache, rmap_item); +} + static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item) { struct mm_struct *mm = rmap_item->mm; @@ -825,7 +860,7 @@ static int unmerge_ksm_pages(struct vm_area_struct *vma, if (signal_pending(current)) err = -ERESTARTSYS; else - err = break_ksm(vma, addr); + err = break_ksm(vma, addr, false); } return err; } @@ -2017,6 +2052,36 @@ static void stable_tree_append(struct ksm_rmap_item *rmap_item, rmap_item->mm->ksm_merging_pages++; } +static int try_to_merge_with_kernel_zero_page(struct mm_struct *mm, + struct ksm_rmap_item *rmap_item, + struct page *page) +{ + int err = 0; + + if (!(rmap_item->address & ZERO_PAGE_FLAG)) { + struct vm_area_struct *vma; + + mmap_read_lock(mm); + vma = find_mergeable_vma(mm, rmap_item->address); + if (vma) { + err = try_to_merge_one_page(vma, page, + ZERO_PAGE(rmap_item->address)); + } else { + /* If the vma is out of date, we do not need to continue. */ + err = 0; + } + mmap_read_unlock(mm); + /* + * In case of failure, the page was not really empty, so we + * need to continue. Otherwise we're done. + */ + if (!err) + rmap_item->address |= ZERO_PAGE_FLAG; + } + + return err; +} + /* * cmp_and_merge_page - first see if page can be merged into the stable tree; * if not, compare checksum to previous and if it's the same, see if page can @@ -2101,29 +2166,21 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite * Same checksum as an empty page. We attempt to merge it with the * appropriate zero page if the user enabled this via sysfs. */ - if (ksm_use_zero_pages && (checksum == zero_checksum)) { - struct vm_area_struct *vma; - - mmap_read_lock(mm); - vma = find_mergeable_vma(mm, rmap_item->address); - if (vma) { - err = try_to_merge_one_page(vma, page, - ZERO_PAGE(rmap_item->address)); - } else { + if (ksm_use_zero_pages) { + if (checksum == zero_checksum) { + /* If success, just return. Otherwise, continue */ + if (!try_to_merge_with_kernel_zero_page(mm, rmap_item, page)) + return; + } else if (rmap_item->address & ZERO_PAGE_FLAG) { /* - * If the vma is out of date, we do not need to - * continue. + * The page now is not kernel zero page (COW happens to it) + * but the flag of its rmap_item is still zero-page, so need + * to reset the flag and update the corresponding count. */ - err = 0; + rmap_item->address &= PAGE_MASK; } - mmap_read_unlock(mm); - /* - * In case of failure, the page was not really empty, so we - * need to continue. Otherwise we're done. - */ - if (!err) - return; } + tree_rmap_item = unstable_tree_search_insert(rmap_item, page, &tree_page); if (tree_rmap_item) { @@ -2337,6 +2394,23 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) mmap_read_unlock(mm); return rmap_item; } + /* + * Because we want to monitor ksm zero pages which is + * non-anonymous, we must try to return the rmap_items + * of those kernel zero pages which replaces its + * original anonymous empty page due to use_zero_pages's + * feature. + */ + if (is_zero_pfn(page_to_pfn(*page))) { + rmap_item = try_to_get_old_rmap_item(ksm_scan.address, + ksm_scan.rmap_list); + if (rmap_item && (rmap_item->address & ZERO_PAGE_FLAG)) { + ksm_scan.rmap_list = &rmap_item->rmap_list; + ksm_scan.address += PAGE_SIZE; + mmap_read_unlock(mm); + return rmap_item; + } + } next_page: put_page(*page); ksm_scan.address += PAGE_SIZE; From patchwork Tue Oct 11 02:22:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: xu xin X-Patchwork-Id: 13003561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 936A8C433F5 for ; Tue, 11 Oct 2022 02:22:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D39C8E0001; Mon, 10 Oct 2022 22:22:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25BB96B0074; Mon, 10 Oct 2022 22:22:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0BB3B8E0001; Mon, 10 Oct 2022 22:22:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E60BC6B0073 for ; Mon, 10 Oct 2022 22:22:37 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B823A402CA for ; Tue, 11 Oct 2022 02:22:37 +0000 (UTC) X-FDA: 80007069954.04.B35171E Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf22.hostedemail.com (Postfix) with ESMTP id 669B6C0028 for ; Tue, 11 Oct 2022 02:22:37 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id g28so12255691pfk.8 for ; Mon, 10 Oct 2022 19:22:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jBGBO38UtGSWwXVp7Ms/n5NhA5Pd+Mi6T8z7EXrrTqE=; b=NQwP6y9bldduXueh4GTe05alGLYH2cXxs6iR6vqImjtYM+ijwRDNkiUFqRfINXVKnc lPNsjMlrdYCD/aQbcS+3J5F6ov+nKAIyIz8fHQnOMnJMVayFdHpk+dUzr0/ShM/aQ4zQ PIYCeXOqyA6j/JdcmzrfsfvNdk95wsVOFdb0rARM1VmPokMffWAPbzgNsgUTBRSW3iQI TGQJmqYdZd5YOSMSmSsSCgLgXogoXIY0JmT/e/olVLdSk5Suu0uIkOGokMlh+UnvnvLC WjzAbxhTg04rIwWiOQcOLB4kfxGpAL7tYhAK4fjuIiOODSi2rgBDazceiLPQFN5OD7rP VPSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jBGBO38UtGSWwXVp7Ms/n5NhA5Pd+Mi6T8z7EXrrTqE=; b=x9hHF2twDdn0IeZOxZzNZJLUNhST/5rhrMIv+D2uj7j4+jwOq0MIN4PZ3vQ82EdNNC HKes9FI97sm9x5iLqYdWgTnVNk1O+kTDdFHpic9781UQvaR68ARI63lmyO1M6heRH4H/ eIiFnN4jDV+RcUEhlvdYCepvYcYGdYvyOBAmhotG7mf4GtPIOyHC0N6B5pOWuvZKzYii rFHww9vfHRqGixlN6lLjFDlZel62Mkkm11+Owgcqjb50WzwHppMh4c6JRnBTm3Z0e/Dn 62V2YEUOMmUfpz44IpSYMfLgjPtmGn83pp7JBEG6nXHLpJCWDbtzAMwga5B3nEE8szpJ HXbw== X-Gm-Message-State: ACrzQf38DsisRaB0b5Jzv+X6U3r5QgdmiBxi5HzgGj8+sPuE8JyB0z+x 3c39O4toZHwb5nMzn+71lbg= X-Google-Smtp-Source: AMsMyM4HJO1bRVw+FEpT/jpF8AW/W1KC5CylFOLFTiM7r3HksmwbDm+KaSDXGiVpS8apzQDl8EckZQ== X-Received: by 2002:a05:6a00:2291:b0:563:9d0d:62ae with SMTP id f17-20020a056a00229100b005639d0d62aemr3557524pfe.17.1665454956397; Mon, 10 Oct 2022 19:22:36 -0700 (PDT) Received: from localhost.localdomain ([193.203.214.57]) by smtp.gmail.com with ESMTPSA id m15-20020a17090a158f00b0020d6fc00072sm69660pja.9.2022.10.10.19.22.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 19:22:35 -0700 (PDT) From: xu.xin.sc@gmail.com X-Google-Original-From: xu.xin16@zte.com.cn To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, xu xin , Claudio Imbrenda , David Hildenbrand , Xuexin Jiang , Xiaokai Ran , Yang Yang Subject: [PATCH v3 3/5] ksm: count all zero pages placed by KSM Date: Tue, 11 Oct 2022 02:22:30 +0000 Message-Id: <20221011022230.322323-1-xu.xin16@zte.com.cn> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221011022006.322158-1-xu.xin16@zte.com.cn> References: <20221011022006.322158-1-xu.xin16@zte.com.cn> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665454957; a=rsa-sha256; cv=none; b=mVXsqQbSwuzyfmO4bokBQGJS07Ax3EOpsQObrdm+AkLsiQSmAKlgC13KthofaX1zdS7ZCa B3u7oO8YWv/UzncTiDeKfAI6+uIQx+zNzS2aYiUKcUculSegGx1OaapV9qiQD4Dml278wO yATGmE2lmHmE3XZaDxg0XJT4dzSwdZU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NQwP6y9b; spf=pass (imf22.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.210.193 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665454957; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jBGBO38UtGSWwXVp7Ms/n5NhA5Pd+Mi6T8z7EXrrTqE=; b=BtpQ48Xg9+dRBwO4ZnvVRZETGf0ArrfRf+MJCS3i0jQRw9kw9ev3XyNFHvVKbUxVcJfsFL PSJIkkiFbyfAxgZoGPSwdeZdPkRsoXHkONLk7cQ/+TT/WfreP2wo4rz25B8HdxFEvzdGjp +ytWjdKuGj2blS3eqGeFRxlrAmT9S+Q= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NQwP6y9b; spf=pass (imf22.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.210.193 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: wshrhost58iddddyhsbemyjnbw86uzjd X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 669B6C0028 X-Rspam-User: X-HE-Tag: 1665454957-388232 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: xu xin As pages_sharing and pages_shared don't include the number of zero pages merged by KSM, we cannot know how many pages are zero pages placed by KSM when enabling use_zero_pages, which leads to KSM not being transparent with all actual merged pages by KSM. In the early days of use_zero_pages, zero-pages was unable to get unshared by the ways like MADV_UNMERGEABLE so it's hard to count how many times one of those zeropages was then unmerged. But now, unsharing KSM-placed zero page accurately has been achieved, so we can easily count both how many times a page full of zeroes was merged with zero-page and how many times one of those pages was then unmerged. and so, it helps to estimate memory demands when each and every shared page could get unshared. So we add zero_pages_sharing under /sys/kernel/mm/ksm/ to show the number of all zero pages placed by KSM. Cc: Claudio Imbrenda Cc: David Hildenbrand Cc: Xuexin Jiang Cc: Xiaokai Ran Cc: Yang Yang Signed-off-by: xu xin --- mm/ksm.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index e351d7b6d15e..2970a7062db6 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -275,6 +275,9 @@ static unsigned int zero_checksum __read_mostly; /* Whether to merge empty (zeroed) pages with actual zero pages */ static bool ksm_use_zero_pages __read_mostly; +/* The number of zero pages placed by KSM use_zero_pages */ +static unsigned long ksm_zero_pages_sharing; + #ifdef CONFIG_NUMA /* Zeroed when merging across nodes is not allowed */ static unsigned int ksm_merge_across_nodes = 1; @@ -542,8 +545,10 @@ static inline int unshare_zero_pages(struct ksm_rmap_item *rmap_item) static inline void free_rmap_item(struct ksm_rmap_item *rmap_item) { - if (rmap_item->address & ZERO_PAGE_FLAG) - unshare_zero_pages(rmap_item); + if (rmap_item->address & ZERO_PAGE_FLAG) { + if (!unshare_zero_pages(rmap_item)) + ksm_zero_pages_sharing--; + } ksm_rmap_items--; rmap_item->mm->ksm_rmap_items--; rmap_item->mm = NULL; /* debug safety */ @@ -2075,8 +2080,10 @@ static int try_to_merge_with_kernel_zero_page(struct mm_struct *mm, * In case of failure, the page was not really empty, so we * need to continue. Otherwise we're done. */ - if (!err) + if (!err) { rmap_item->address |= ZERO_PAGE_FLAG; + ksm_zero_pages_sharing++; + } } return err; @@ -2178,6 +2185,7 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite * to reset the flag and update the corresponding count. */ rmap_item->address &= PAGE_MASK; + ksm_zero_pages_sharing--; } } @@ -3190,6 +3198,13 @@ static ssize_t pages_volatile_show(struct kobject *kobj, } KSM_ATTR_RO(pages_volatile); +static ssize_t zero_pages_sharing_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%ld\n", ksm_zero_pages_sharing); +} +KSM_ATTR_RO(zero_pages_sharing); + static ssize_t stable_node_dups_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -3250,6 +3265,7 @@ static struct attribute *ksm_attrs[] = { &merge_across_nodes_attr.attr, #endif &max_page_sharing_attr.attr, + &zero_pages_sharing_attr.attr, &stable_node_chains_attr.attr, &stable_node_dups_attr.attr, &stable_node_chains_prune_millisecs_attr.attr, From patchwork Tue Oct 11 02:22:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: xu xin X-Patchwork-Id: 13003562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB27DC433F5 for ; Tue, 11 Oct 2022 02:22:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78F7C8E0001; Mon, 10 Oct 2022 22:22:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 716446B0074; Mon, 10 Oct 2022 22:22:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56A628E0001; Mon, 10 Oct 2022 22:22:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3AC9E6B0073 for ; Mon, 10 Oct 2022 22:22:54 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 01C9D160814 for ; Tue, 11 Oct 2022 02:22:53 +0000 (UTC) X-FDA: 80007070668.20.01CDEC4 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) by imf27.hostedemail.com (Postfix) with ESMTP id 9A6E14001A for ; Tue, 11 Oct 2022 02:22:53 +0000 (UTC) Received: by mail-pj1-f68.google.com with SMTP id p3-20020a17090a284300b0020a85fa3ffcso14655154pjf.2 for ; Mon, 10 Oct 2022 19:22:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0i90jsiDje3OippwaWc7d+5Gj/jeYy2WnHfwavKx1Zc=; b=MbQSbPtXvqYlFl1N+H0dMvFlCM2YQF26s9pH/nam6VIgQ2wFgvKeUspQIecinHRbUs Hj9fALXGL82tlCQZCVhYowBU8NZsqc7tDHR7jptur05LWdJ9IRwsjK4tPmvC9LlYRxNr K87nnfFq8DMfBcx9NkiACVW1/uAcUwQ6zoOmr5pPYeNkV9XHunRzCPT+Tktu/drkcLt6 XD6UV/aXHyM686jv87C2eBwtIvvZHSepGPXufc4227d2Em3NzYVP+7cJX59xpguD/Hpn ZgUtsxjWQ8qqRzFCHaF6RYD3lqNp64V+Kg38o3fnJ1bYiKBjdfaagRf8YzMw9q09zHVm 6pPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0i90jsiDje3OippwaWc7d+5Gj/jeYy2WnHfwavKx1Zc=; b=1t/jeqiHZlDxJasKe9iTRazl7llmnPgwWsgeKAvkMrhd0uAMJsNxKN8VdYVjFFwNH9 56wTAo2CdvL2APIWkKlGxsd6jGCJHT6GJs4OjPyn//foggCWEwqy8QffsGPLI9vHFF+A UVRWsGmMfMLrqadkouxfeRtDBBcVa4ZyNuCnIJiOE4tz4M96uYtJ7L1qHjvWKPHoCQoN mDQfRFxM51H89rtpN+7R26dBV75GHNRLHkiJmqW3i4mp6EgBGlDtJSZLTT670Dbxii3f HXeQjx2gKR6VXI3lfkP/uFer2UvCYSxma8t+e+MJpi0KQAjJeQ8aQfHaa8uaSnqyjKEn U8mg== X-Gm-Message-State: ACrzQf0YYtZTMTRpTVT0VgFGzX+BLINHYyc0qVryQVs0yFYxgbEbslJx adJeeXGne+rW8spW5iDK0Ug= X-Google-Smtp-Source: AMsMyM4z5grjzv3Vvk1EW21MJh3qJtZLK69EMl0iS6JtR43bWHIz9+nVBnfZX/qCphFVvPIBr6OMSw== X-Received: by 2002:a17:902:dad1:b0:183:243c:d0d0 with SMTP id q17-20020a170902dad100b00183243cd0d0mr4632808plx.157.1665454972670; Mon, 10 Oct 2022 19:22:52 -0700 (PDT) Received: from localhost.localdomain ([193.203.214.57]) by smtp.gmail.com with ESMTPSA id e2-20020aa798c2000000b005631a40a00bsm4423241pfm.139.2022.10.10.19.22.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 19:22:52 -0700 (PDT) From: xu.xin.sc@gmail.com X-Google-Original-From: xu.xin16@zte.com.cn To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, xu xin , Claudio Imbrenda , David Hildenbrand , Xuexin Jiang , Xiaokai Ran , Yang Yang Subject: [PATCH v3 4/5] ksm: count zero pages for each process Date: Tue, 11 Oct 2022 02:22:46 +0000 Message-Id: <20221011022246.322377-1-xu.xin16@zte.com.cn> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221011022006.322158-1-xu.xin16@zte.com.cn> References: <20221011022006.322158-1-xu.xin16@zte.com.cn> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665454973; a=rsa-sha256; cv=none; b=sDDyh7Fv+xcORWwvHg89KwwvAy/9K8T7Sdqb1uiJE7DgFRwg2Vhso8gEDZgrJwB9f5YFns GArKQOsXtKuhx/VybTC9RZT8Vv5cadFm7CUhYhWx+6tML7V/UCdCjax8pXy3EZkpWYWOOD LGSgbrho/0NPsAyaIIgo9hs0RH+H3M8= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MbQSbPtX; spf=pass (imf27.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.216.68 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665454973; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0i90jsiDje3OippwaWc7d+5Gj/jeYy2WnHfwavKx1Zc=; b=ruXFcrc5tCKTPjIJf+UT9aZ4HSYmZ0BTwXLDqmcwWox0S9m4HQUTihsIMt75f1vsVF14IK mVARtnGqC+jR9QlEBEw67KP1h6vUicq+Lt+X9dAw3AFFsyyyEperIQisttJr6X82h8totD QfuEOCX0ofNqrin1bZ75CVKJb9zQLSA= Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MbQSbPtX; spf=pass (imf27.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.216.68 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: apq47pcg5buyy4musgpkm11iyyzazy5g X-Rspamd-Queue-Id: 9A6E14001A X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1665454973-594104 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: xu xin As the number of ksm zero pages is not included in ksm_merging_pages per process when enabling use_zero_pages, it's unclear of how many actual pages are merged by KSM. To let users accurately estimate their memory demands when unsharing KSM zero-pages, it's necessary to show KSM zero- pages per process. since unsharing zero pages placed by KSM accurately is achieved, then tracking empty pages merging and unmerging is not a difficult thing any longer. Since we already have /proc//ksm_stat, just add the information of zero_pages_sharing in it. Cc: Claudio Imbrenda Cc: David Hildenbrand Cc: Xuexin Jiang Cc: Xiaokai Ran Cc: Yang Yang Signed-off-by: xu xin --- fs/proc/base.c | 1 + include/linux/mm_types.h | 7 ++++++- mm/ksm.c | 6 +++++- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/fs/proc/base.c b/fs/proc/base.c index 9e479d7d202b..ac9ebe972be0 100644 --- a/fs/proc/base.c +++ b/fs/proc/base.c @@ -3207,6 +3207,7 @@ static int proc_pid_ksm_stat(struct seq_file *m, struct pid_namespace *ns, mm = get_task_mm(task); if (mm) { seq_printf(m, "ksm_rmap_items %lu\n", mm->ksm_rmap_items); + seq_printf(m, "zero_pages_sharing %lu\n", mm->ksm_zero_pages_sharing); mmput(mm); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 500e536796ca..78a4ee264645 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -691,7 +691,7 @@ struct mm_struct { #ifdef CONFIG_KSM /* * Represent how many pages of this process are involved in KSM - * merging. + * merging (not including ksm_zero_pages_sharing). */ unsigned long ksm_merging_pages; /* @@ -699,6 +699,11 @@ struct mm_struct { * including merged and not merged. */ unsigned long ksm_rmap_items; + /* + * Represent how many empty pages are merged with kernel zero + * pages when enabling KSM use_zero_pages. + */ + unsigned long ksm_zero_pages_sharing; #endif #ifdef CONFIG_LRU_GEN struct { diff --git a/mm/ksm.c b/mm/ksm.c index 2970a7062db6..c049a95afc26 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -546,8 +546,10 @@ static inline int unshare_zero_pages(struct ksm_rmap_item *rmap_item) static inline void free_rmap_item(struct ksm_rmap_item *rmap_item) { if (rmap_item->address & ZERO_PAGE_FLAG) { - if (!unshare_zero_pages(rmap_item)) + if (!unshare_zero_pages(rmap_item)) { ksm_zero_pages_sharing--; + rmap_item->mm->ksm_zero_pages_sharing--; + } } ksm_rmap_items--; rmap_item->mm->ksm_rmap_items--; @@ -2083,6 +2085,7 @@ static int try_to_merge_with_kernel_zero_page(struct mm_struct *mm, if (!err) { rmap_item->address |= ZERO_PAGE_FLAG; ksm_zero_pages_sharing++; + rmap_item->mm->ksm_zero_pages_sharing++; } } @@ -2186,6 +2189,7 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite */ rmap_item->address &= PAGE_MASK; ksm_zero_pages_sharing--; + rmap_item->mm->ksm_zero_pages_sharing--; } } From patchwork Tue Oct 11 02:23:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: xu xin X-Patchwork-Id: 13003563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37ECAC433FE for ; Tue, 11 Oct 2022 02:23:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD36E6B0072; Mon, 10 Oct 2022 22:23:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B5B4A6B0073; Mon, 10 Oct 2022 22:23:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9ADCE8E0001; Mon, 10 Oct 2022 22:23:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 80E5D6B0072 for ; Mon, 10 Oct 2022 22:23:39 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 551E6A046D for ; Tue, 11 Oct 2022 02:23:39 +0000 (UTC) X-FDA: 80007072558.15.050C285 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by imf19.hostedemail.com (Postfix) with ESMTP id F080E1A0021 for ; Tue, 11 Oct 2022 02:23:38 +0000 (UTC) Received: by mail-pl1-f194.google.com with SMTP id c24so11915113pls.9 for ; Mon, 10 Oct 2022 19:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pA2L/FNaj7e7+kUZGmo87Y5GGtuU8PgqF9oHvyd5XJ0=; b=aOVAgCTFUk3B94yRy18snaZHgtEeSUJykJ0ot+Ljr1hdS5x9wwPdboGW7YMhvDAbRJ ZwvfxbttriYA3H2esaRMMKmVQ0oj9gP02KjMIUTv8viD6oOBtwmpwO/Zmz+LYVUu0dA9 +sF6lCBmK1N0DRiU09vUW/bOP4i8SB9FupMR+m1BQzVXU8kxf29K79Wl1HZWJHmgeKZx oDZWo8snoo2zIPjyDZX96GeLSt4KLsiT/ffMlszxDQI61qJ/iHY8p1guKTIdUMpXoBJa XDqHWaFPlvXmZ2BxY/Iq+yDHZPbPWeLWzwuIc5DzGIUKzmMXwRSjZsNVwgxcB3f6B7Of YUFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pA2L/FNaj7e7+kUZGmo87Y5GGtuU8PgqF9oHvyd5XJ0=; b=6E6KWAK5tVBWGrTOLgLWQo9mL0tEfLmo9LTjiny1/Rs2GPddkZyEI1Hn+AXR+Rx82d V1z0GI//AIvQADw2uyN+KNJEaGdBJn9cS9eCDEKZ6TY8Ng4EoPaL/2XLUb88Gg32ZQr+ ZK/gNXN9ylQVRPKFc2kfT3mFeFZw7pAZT5jZOlb8qwTTY1tWkg2+kfQRh0cEsCEu+qNQ Eieukn33S2OUKl8g+vq67R2/RqiuGGiGHB1WLau8ObJHCefQwyUyLpzkX94oH9zgLqbA Qt/XSG+JrIUgxAPawx6CPdjNL41LaSg1APOKV3BFk4oTxnjdi0NnQw9H6o0+E1jAoO2Q PN8A== X-Gm-Message-State: ACrzQf2P1dmqIuQV2CJJ/wHyeShKT8+L/HAwDrzq2vSHl0WxcgXVW5aH DrB7+k0M4lZU20ODlahgMgc= X-Google-Smtp-Source: AMsMyM6Fv/99kgTYy5VGB7ccfHNOf0+TN7/WFuO0wEJREVgxfoyTXj54YlueWcfwwlxOs9NAHq3MQQ== X-Received: by 2002:a17:90a:ac11:b0:20b:10d2:e837 with SMTP id o17-20020a17090aac1100b0020b10d2e837mr24508899pjq.165.1665455018060; Mon, 10 Oct 2022 19:23:38 -0700 (PDT) Received: from localhost.localdomain ([193.203.214.57]) by smtp.gmail.com with ESMTPSA id h2-20020a63f902000000b00438834b14a1sm6801151pgi.80.2022.10.10.19.23.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 19:23:37 -0700 (PDT) From: xu.xin.sc@gmail.com X-Google-Original-From: xu.xin16@zte.com.cn To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, xu xin , Xiaokai Ran , Yang Yang , Jiang Xuexin , Claudio Imbrenda , David Hildenbrand Subject: [PATCH v3 5/5] ksm: add zero_pages_sharing documentation Date: Tue, 11 Oct 2022 02:23:32 +0000 Message-Id: <20221011022332.322435-1-xu.xin16@zte.com.cn> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221011022006.322158-1-xu.xin16@zte.com.cn> References: <20221011022006.322158-1-xu.xin16@zte.com.cn> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665455019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pA2L/FNaj7e7+kUZGmo87Y5GGtuU8PgqF9oHvyd5XJ0=; b=4ikJ+lpbe+fpY1xJJQA3X3nMb4UW4HYTM9SZJiUv4NgUfHaUmYmrekddC5GDhm7oPUmYNk dZ7wO0ofWgJOPj1ZmXpW5ZaVfoWzfjtmCJPEEqcEP2MHrVoNV922R4j5o5LbYIB0Qj5xeK hN/OFAxnp3k2pIdoF8d3m4sAWMuE2l0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aOVAgCTF; spf=pass (imf19.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.214.194 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665455019; a=rsa-sha256; cv=none; b=XSvZlj2A7ba3HNGp3zhxdcoZvyKusCb351OPbT2ffo1j+ktdQDYft68fCJX5yVf9jihwgM T5WT/EwzLZ1V0PbEi07ld9Ux+h+l/E9S0npXXjMlfhlSvHkaLr+CqvrhQc6koN8+MYZUdV hhOrV1jDpIU7/KLWx8DPDM0GJCOhVXs= Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aOVAgCTF; spf=pass (imf19.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.214.194 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: gj4k4sqyanazxhx9siirjfizwjxrq6or X-Rspamd-Queue-Id: F080E1A0021 X-HE-Tag: 1665455018-603359 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: xu xin When enabling use_zero_pages, pages_sharing cannot represent how much memory saved indeed. zero_pages_sharing + pages_sharing does. add the description of zero_pages_sharing. Cc: Xiaokai Ran Cc: Yang Yang Cc: Jiang Xuexin Cc: Claudio Imbrenda Cc: David Hildenbrand Signed-off-by: xu xin --- Documentation/admin-guide/mm/ksm.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/mm/ksm.rst b/Documentation/admin-guide/mm/ksm.rst index fb6ba2002a4b..484665aa7418 100644 --- a/Documentation/admin-guide/mm/ksm.rst +++ b/Documentation/admin-guide/mm/ksm.rst @@ -162,7 +162,7 @@ The effectiveness of KSM and MADV_MERGEABLE is shown in ``/sys/kernel/mm/ksm/``: pages_shared how many shared pages are being used pages_sharing - how many more sites are sharing them i.e. how much saved + how many more sites are sharing them pages_unshared how many pages unique but repeatedly checked for merging pages_volatile @@ -173,6 +173,14 @@ stable_node_chains the number of KSM pages that hit the ``max_page_sharing`` limit stable_node_dups number of duplicated KSM pages +zero_pages_sharing + how many empty pages are sharing kernel zero page(s) instead of + with each other as it would happen normally. Only effective when + enabling ``use_zero_pages`` knob. + +If ``use_zero_pages`` is 0, only ``pages_sharing`` can represents how +much saved. Otherwise, ``pages_sharing`` + ``zero_pages_sharing`` +represents how much saved actually. A high ratio of ``pages_sharing`` to ``pages_shared`` indicates good sharing, but a high ratio of ``pages_unshared`` to ``pages_sharing``