From patchwork Sun Oct 9 02:23:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: xu xin X-Patchwork-Id: 13001879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84128C433FE for ; Sun, 9 Oct 2022 02:23:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03DF96B0071; Sat, 8 Oct 2022 22:23:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F08206B0073; Sat, 8 Oct 2022 22:23:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5AAD6B0074; Sat, 8 Oct 2022 22:23:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BBCEA6B0071 for ; Sat, 8 Oct 2022 22:23:11 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CA2E9A0168 for ; Sun, 9 Oct 2022 02:23:10 +0000 (UTC) X-FDA: 79999813740.01.2D2D6F3 Received: from mail-pj1-f66.google.com (mail-pj1-f66.google.com [209.85.216.66]) by imf09.hostedemail.com (Postfix) with ESMTP id 6FC2A140021 for ; Sun, 9 Oct 2022 02:23:10 +0000 (UTC) Received: by mail-pj1-f66.google.com with SMTP id 70so7287923pjo.4 for ; Sat, 08 Oct 2022 19:23:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pZ2+yi+8Cy6ea/BvAsMEkhQ4pii2mF27mTTm96t3VBo=; b=hmVN/VbmXsVIca+O9lSyXl7yVnjrBsRWpjgJh7tztjuDDFNe79jgL/7ENKZgE6W75Q zAFpLmIpf/sc9MHxVNyaHdXqpmgPaUGswY+T1o0dV9lZ0kBpbhhJqMtYPZzBehSurJch 9w80NOemig3XcODDC3RziJnf9GYfixChHHAhTNNBKvO6bAuNjC9uMAqliARaj3VERlxO ApTFtUQzITehlx0mqkd9eFETXnIHCFRSPcsVM2Ak7BM+UkvgHi9gWn+nFOKWgVc6Qhsb blleOhlmtqyLcN9c9HdU6QDuIziWV8DmLQKPU0LblzZaCTaxip4SHbEkXiAuW3PpjRnj 7dGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pZ2+yi+8Cy6ea/BvAsMEkhQ4pii2mF27mTTm96t3VBo=; b=CRyyKYcbs1kBfcI/oQp86M7NX2M1zw63lYWNb0Yfl5vhkxkePBLrSSzvODLgE76keg On3mUbhfBO9gySFF1RzS5BJ7dlzGPmbL3E7absswuO8dkZ7Z0Gig9ewSBaLUvmfCBbSp hb0zS04XGglbcTQ9YZ6QcXK0hWiZBPAOT78Jsq0RU0EuWouZK6qvunz+wWM7aemdD+lm gKENYz1mdhAbdPLgbXix2KN64huUaUn1i9trfjqntrdmPQeieN578LE3reUhnDKX0CKp RgrzEOsxAADpGdx3PrB+gtiDeDRcF1S8KQ0ZgciEwAzpkH0N4UIY/8TrsaP16/+nJa6S jQMw== X-Gm-Message-State: ACrzQf2INECoOAf8faX8dzyzC1nyN83eJF8Hi3LK4gk46GZX98CwX4C9 Lo9FME/W9yBXcCbXMSgpVHU= X-Google-Smtp-Source: AMsMyM5ftPPoFEo1quN0Z6hL33p1YRq2w6VQdOOo9gynaqnyRjqAWa2xBVxLRplXTS+62wc57oO8Fw== X-Received: by 2002:a17:90a:4607:b0:202:e22d:489c with SMTP id w7-20020a17090a460700b00202e22d489cmr24665392pjg.80.1665282189349; Sat, 08 Oct 2022 19:23:09 -0700 (PDT) Received: from localhost.localdomain ([193.203.214.57]) by smtp.gmail.com with ESMTPSA id b10-20020a170902d50a00b0016dc6279ab7sm1737835plg.149.2022.10.08.19.23.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 08 Oct 2022 19:23:08 -0700 (PDT) From: xu.xin.sc@gmail.com X-Google-Original-From: xu.xin16@zte.com.cn To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, xu xin , David Hildenbrand , Claudio Imbrenda , Xuexin Jiang , Xiaokai Ran , Yang Yang Subject: [PATCH v2 2/5] ksm: support unsharing zero pages placed by KSM Date: Sun, 9 Oct 2022 02:23:02 +0000 Message-Id: <20221009022302.315318-1-xu.xin16@zte.com.cn> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221009021816.315205-1-xu.xin16@zte.com.cn> References: <20221009021816.315205-1-xu.xin16@zte.com.cn> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665282190; a=rsa-sha256; cv=none; b=dpSnVC8zjYC0peTG5FrfPeGiASOM1CiAwhV/1vQojQgx7pQaKTOcblxi1snnKF3GcXrw1i jDeTcHJlfaZx4od3VKcEzU5EfdUE16V3o2514qAUy5N9NbmjKTt84qmZyycCj7kBarjgxE 2ncVQpwACmqWRPVos1XGLAnswMhtQBQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="hmVN/Vbm"; spf=pass (imf09.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.216.66 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665282190; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pZ2+yi+8Cy6ea/BvAsMEkhQ4pii2mF27mTTm96t3VBo=; b=l9JS2fb0Lghtz3JIL3tGfRQapRRCFHInYYPOXoBxSaRLgT/TARlq8spqsBioOQAxFXFrRt YpxMMYwYn/1UP6zgcNGGA9u/B7uH02o/SGPsRBpvflxI3C9e/0dtnE/YO52S11wEcJd6e7 qtSYclgZ2yS1nvn+fnsrf+O9d7Ri8ps= Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="hmVN/Vbm"; spf=pass (imf09.hostedemail.com: domain of xu.xin.sc@gmail.com designates 209.85.216.66 as permitted sender) smtp.mailfrom=xu.xin.sc@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: e69z6k5urenbzk6phgehgccwziojsue4 X-Rspamd-Queue-Id: 6FC2A140021 X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1665282190-843545 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: xu xin After the commit e86c59b1b12d ("mm/ksm: improve deduplication of zero pages with colouring"), madvise(addr, len, MADV_UNMERGEABLE) and other ways (like write 2 to /sys/kernel/mm/ksm/run) to trigger unsharing will **not** unshare the shared zeropage as placed by KSM (which is against the MADV_UNMERGEABLE documentation at least). To not blindly unshare all shared zero_pages in applicable VMAs, the patch introduces a dedicated flag ZERO_PAGE_FLAG to mark the rmap_items of those shared zero_pages. and guarantee that these rmap_items will be not freed during the time of zero_pages not being writing, so we can only unshare the *KSM-placed* zero_pages. The patch will not degrade the performance of use_zero_pages as it doesn't change the way of merging empty pages in use_zero_pages's feature. Fixes: e86c59b1b12d ("mm/ksm: improve deduplication of zero pages with colouring") Reported-by: David Hildenbrand Cc: Claudio Imbrenda Cc: Xuexin Jiang Signed-off-by: xu xin Co-developed-by: Xiaokai Ran Signed-off-by: Xiaokai Ran Co-developed-by: Yang Yang Signed-off-by: Yang Yang Signed-off-by: xu xin --- mm/ksm.c | 133 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 103 insertions(+), 30 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 13c60f1071d8..e988a17b837e 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -213,6 +213,7 @@ struct ksm_rmap_item { #define SEQNR_MASK 0x0ff /* low bits of unstable tree seqnr */ #define UNSTABLE_FLAG 0x100 /* is a node of the unstable tree */ #define STABLE_FLAG 0x200 /* is listed from the stable tree */ +#define ZERO_PAGE_FLAG 0x400 /* is zero page placed by KSM */ /* The stable and unstable tree heads */ static struct rb_root one_stable_tree[1] = { RB_ROOT }; @@ -381,14 +382,6 @@ static inline struct ksm_rmap_item *alloc_rmap_item(void) return rmap_item; } -static inline void free_rmap_item(struct ksm_rmap_item *rmap_item) -{ - ksm_rmap_items--; - rmap_item->mm->ksm_rmap_items--; - rmap_item->mm = NULL; /* debug safety */ - kmem_cache_free(rmap_item_cache, rmap_item); -} - static inline struct ksm_stable_node *alloc_stable_node(void) { /* @@ -434,7 +427,8 @@ static inline bool ksm_test_exit(struct mm_struct *mm) * of the process that owns 'vma'. We also do not want to enforce * protection keys here anyway. */ -static int break_ksm(struct vm_area_struct *vma, unsigned long addr) +static int break_ksm(struct vm_area_struct *vma, unsigned long addr, + bool ksm_check_bypass) { struct page *page; vm_fault_t ret = 0; @@ -449,6 +443,16 @@ static int break_ksm(struct vm_area_struct *vma, unsigned long addr) ret = handle_mm_fault(vma, addr, FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, NULL); + else if (ksm_check_bypass && is_zero_pfn(page_to_pfn(page))) { + /* + * Although it's not ksm page, it's zero page as placed by + * KSM use_zero_page, so we should unshare it when + * ksm_check_bypass is true. + */ + ret = handle_mm_fault(vma, addr, + FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE, + NULL); + } else ret = VM_FAULT_WRITE; put_page(page); @@ -496,6 +500,11 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm, return vma; } +/* + * Note: Don't call break_cow() in the context protected by + * mmap_read_lock(), which may cause dead lock because inside + * break_cow mmap_read_lock exists. + */ static void break_cow(struct ksm_rmap_item *rmap_item) { struct mm_struct *mm = rmap_item->mm; @@ -511,10 +520,35 @@ static void break_cow(struct ksm_rmap_item *rmap_item) mmap_read_lock(mm); vma = find_mergeable_vma(mm, addr); if (vma) - break_ksm(vma, addr); + break_ksm(vma, addr, false); mmap_read_unlock(mm); } +/* Only called when rmap_item->address is with ZERO_PAGE_FLAG */ +static inline int unshare_zero_pages(struct ksm_rmap_item *rmap_item) +{ + struct mm_struct *mm = rmap_item->mm; + struct vm_area_struct *vma; + unsigned long addr = rmap_item->address; + int err = -EFAULT; + + vma = vma_lookup(mm, addr); + if (vma) + err = break_ksm(vma, addr, true); + + return err; +} + +static inline void free_rmap_item(struct ksm_rmap_item *rmap_item) +{ + if (rmap_item->address & ZERO_PAGE_FLAG) + unshare_zero_pages(rmap_item); + ksm_rmap_items--; + rmap_item->mm->ksm_rmap_items--; + rmap_item->mm = NULL; /* debug safety */ + kmem_cache_free(rmap_item_cache, rmap_item); +} + static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item) { struct mm_struct *mm = rmap_item->mm; @@ -825,7 +859,7 @@ static int unmerge_ksm_pages(struct vm_area_struct *vma, if (signal_pending(current)) err = -ERESTARTSYS; else - err = break_ksm(vma, addr); + err = break_ksm(vma, addr, NULL); } return err; } @@ -2017,6 +2051,36 @@ static void stable_tree_append(struct ksm_rmap_item *rmap_item, rmap_item->mm->ksm_merging_pages++; } +static int try_to_merge_with_kernel_zero_page(struct mm_struct *mm, + struct ksm_rmap_item *rmap_item, + struct page *page) +{ + int err = 0; + + if (!(rmap_item->address & ZERO_PAGE_FLAG)) { + struct vm_area_struct *vma; + + mmap_read_lock(mm); + vma = find_mergeable_vma(mm, rmap_item->address); + if (vma) { + err = try_to_merge_one_page(vma, page, + ZERO_PAGE(rmap_item->address)); + } else { + /* If the vma is out of date, we do not need to continue. */ + err = 0; + } + mmap_read_unlock(mm); + /* + * In case of failure, the page was not really empty, so we + * need to continue. Otherwise we're done. + */ + if (!err) + rmap_item->address |= ZERO_PAGE_FLAG; + } + + return err; +} + /* * cmp_and_merge_page - first see if page can be merged into the stable tree; * if not, compare checksum to previous and if it's the same, see if page can @@ -2101,29 +2165,21 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite * Same checksum as an empty page. We attempt to merge it with the * appropriate zero page if the user enabled this via sysfs. */ - if (ksm_use_zero_pages && (checksum == zero_checksum)) { - struct vm_area_struct *vma; - - mmap_read_lock(mm); - vma = find_mergeable_vma(mm, rmap_item->address); - if (vma) { - err = try_to_merge_one_page(vma, page, - ZERO_PAGE(rmap_item->address)); - } else { + if (ksm_use_zero_pages) { + if (checksum == zero_checksum) { + /* If success, just return. Otherwise, continue */ + if (!try_to_merge_with_kernel_zero_page(mm, rmap_item, page)) + return; + } else if (rmap_item->address & ZERO_PAGE_FLAG) { /* - * If the vma is out of date, we do not need to - * continue. + * The page now is not kernel zero page (COW happens to it) + * but the flag of its rmap_item is still zero-page, so need + * to reset the flag and update the corresponding count. */ - err = 0; + rmap_item->address &= PAGE_MASK; } - mmap_read_unlock(mm); - /* - * In case of failure, the page was not really empty, so we - * need to continue. Otherwise we're done. - */ - if (!err) - return; } + tree_rmap_item = unstable_tree_search_insert(rmap_item, page, &tree_page); if (tree_rmap_item) { @@ -2337,6 +2393,23 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) mmap_read_unlock(mm); return rmap_item; } + /* + * Because we want to monitor ksm zero pages which is + * non-anonymous, we must try to return the rmap_items + * of those kernel zero pages which replaces its + * original anonymous empty page due to use_zero_pages's + * feature. + */ + if (is_zero_pfn(page_to_pfn(*page))) { + rmap_item = try_to_get_old_rmap_item(ksm_scan.address, + ksm_scan.rmap_list); + if (rmap_item && (rmap_item->address & ZERO_PAGE_FLAG)) { + ksm_scan.rmap_list = &rmap_item->rmap_list; + ksm_scan.address += PAGE_SIZE; + mmap_read_unlock(mm); + return rmap_item; + } + } next_page: put_page(*page); ksm_scan.address += PAGE_SIZE;