From patchwork Sat Feb 18 00:27:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13145389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14313C05027 for ; Sat, 18 Feb 2023 00:29:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FC07280013; Fri, 17 Feb 2023 19:29:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AB89280002; Fri, 17 Feb 2023 19:29:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01212280013; Fri, 17 Feb 2023 19:29:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E87DE280002 for ; Fri, 17 Feb 2023 19:29:08 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BE2C4AAF95 for ; Sat, 18 Feb 2023 00:29:08 +0000 (UTC) X-FDA: 80478527976.20.8B38CED Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id 1765840006 for ; Sat, 18 Feb 2023 00:29:06 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eyB1wewX; spf=pass (imf27.hostedemail.com: domain of 30hvwYwoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=30hvwYwoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676680147; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LnPCXYu0bp3jbi4b+RX4dTJj+ODlc15FNHREFY5xs1M=; b=M8irPkmpWvqp6PIFKtdo3TRNHqUuU0zhGpr8itTe+jLU5ZGNelQyZGXMv0bJyVuVkFhqWZ gMOGZNL73I/hgPOduHUchWzihiEmfLcglpkyODJV1NoOVel22UhbgYVZTxkrm9O8QfAp5C qGZq+j2BIOmcc0/XOOMP4cURNyALW+A= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eyB1wewX; spf=pass (imf27.hostedemail.com: domain of 30hvwYwoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=30hvwYwoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676680147; a=rsa-sha256; cv=none; b=wzLBDfxnjNpS31pKIqoQgBW4/65IQoP+C3AnwdEyQfEaXSR26fq4XAlfs+MqwoMLO8qss0 kDxNyNgJvIl7vF8InpoP7yPc8X6XAntzvSFVoQvaNImocHvB666QXdPft7+RC+AZJLN3w8 X/7jfH+9t2gqqtAlJ73IPw8evKYykz8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536629d5255so17213977b3.2 for ; Fri, 17 Feb 2023 16:29:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LnPCXYu0bp3jbi4b+RX4dTJj+ODlc15FNHREFY5xs1M=; b=eyB1wewXCAx1HDERPMCO0zwZAas4YhYFgV+T58Xo/niWsDRG5UC3Rs6XOZmH1VN6an BDeTgeZmAtV2U/RJBmPsaw2Pzs+502TrDV5JW7uSQjIUphjyqQ6JBLVLZiSywnbCwmN7 glxseLDjZd3DHe25r9gB54clLLLOql2zDYooJkeUUmIQE39+XJaDt5UKqvRy0vPMNP/Y dlrZCv9EsYeOE222dzKUD7vcvUi3Z842TugU3j8cUBo6f42mvUEeN6sqIIjBwYUd3WND 0vcWoKQXj/p0vdSibSMZSUoLzsYHkjTdgECKTuMGuOdbRhA9Y6lS7gs45B0ytQxsj3iT jZcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LnPCXYu0bp3jbi4b+RX4dTJj+ODlc15FNHREFY5xs1M=; b=0JKFiLt4+pQ8S/Cj+GAtwklyHrfs5CksZZdA+E9rcjMaYO1Stn/iQtOCD8CqVKdLhE SU4Kl/5qFjrE8ewMLQRyUET7Nar8kTU/V+ZcHZv4efpYC49qOsZNfuE4CTwbhIndeQIn zW0HswXFRu08unYH0T9yLXQv0eQk/qLwpMNWB+Pai+fzlEc7Z34OsKD4Tx1Dj8jHozm+ aI+yrvcoCBG+mqh8bgDQNb79z4r5aeSKFg4aKHfOkE4fCZWnapgqI6nJ37q11g0I4Tfw vl/aJVDlmZWYKxeh+Pwi1elWYhBaZ1goYDpG2B/YR5shuW0ax3Eh95Pq2LIRUdHCrhYd Vp6Q== X-Gm-Message-State: AO0yUKW3WMQsVNbKFKl+W9HhpNxACDUUsxCFZc/5O5RrjiL6mQNXalks 0wvzRNV1JhknFkbbTW5iipzFTTpG31uV00GD X-Google-Smtp-Source: AK7set8Hmr9k0Rf2FDN/phGApjkx1AsdUQUwGd2TSqsgrixKWblzVEB77ByAXj7RTzk+3xBvLZoYclWlE/0WLKho X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a5b:ec7:0:b0:965:bac9:d458 with SMTP id a7-20020a5b0ec7000000b00965bac9d458mr8139ybs.11.1676680146246; Fri, 17 Feb 2023 16:29:06 -0800 (PST) Date: Sat, 18 Feb 2023 00:27:56 +0000 In-Reply-To: <20230218002819.1486479-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230218002819.1486479-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog Message-ID: <20230218002819.1486479-24-jthoughton@google.com> Subject: [PATCH v2 23/46] hugetlb: add HGM support to move_hugetlb_page_tables From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu , Andrew Morton Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Frank van der Linden , Jiaqi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Stat-Signature: tq57y16puobqfrqktr65zdxpgmyiaqbn X-Rspam-User: X-Rspamd-Queue-Id: 1765840006 X-Rspamd-Server: rspam06 X-HE-Tag: 1676680146-920110 X-HE-Meta: U2FsdGVkX1//JBTE2hPxbXd7/J4mNy65AYtk++0SsmBOLPsVp70SrZZE50ouxoMhcaPcRqWAhOrilF1nSY/g9BypoVj6ELeeL1xv150H9qcMINwGSJ9JfiQZosKhkl0vBJazL+7WuvDR6iaf/0wmUYUiBmgQxm/hbGD0utpzUdeJEFzhqqXup5sLk/wLGfjBtoQ6kNihvGiFo2amBOJ9Kse6uQzLkJaYMXe3BW7369GzBbTvvaYFxRvFTicq1rnID9VWTyvPt979GS400hsAXFWfjpUhkpznrSJoyTxzVfFzQZUDLIDx0ZLkZmgrj74DgSPy9Tum2Kx7GOyEs3/kZCN5fqJfZo8juOCnxuo+xixBfYsgAKh1tajKRDxxKWiCSrnrMP0IcrZ2BmvRueQWb6SBWwF8k5NFgt/wCkxW21fSP/uikKf37qOpBHBUhmRG5F47hdzj1RGLhFZ21d/eEAMHahgIZgPtoNegpqGcMLyS78ytVwp4Twz8URyNf0I782IdxQyCHnEG01wboljxgyTR61DwM0RFRFWwAnZSAGpht9AkPP/H3gopf9Z5I/gk9Mx9i7TqP4Ul/Adln+BQfbvCFqKFvIY0T7bMsySRIKuMn8qMv6MpgyP2NM6javluaio2a7YJ9K0BmbkYsW5SYI1H+Dr9nz3oJxFAd9jLt6zHatoc00GZoRKOwBANgnu4wtUa5l2SBDpIsO2W9sTkfxtet/mhtX8j77nPBCoEEuO8Yh4lcYPD8EXPz7ieKMWGoR+nWdFwpzjDGyq+pCw4Y9u/ikmHAQAUjudNO47iQLQJl/N63p96DAedlJE9FLJHKDXbxgZPzTDIqx9sXSwKsuYLPM4mWSsLO6E0EotdHt9+GNoytkd8E5XpAJkLeKabCLdI6Q/nUWOLOmO+yM1V6sar9pQ6CK+XxuI2kfrL7/c2pcY6JEMc7cBf7O9b/nTpKl5kaNgoc3OPu07mywE GM+09UmR rhkFjDUQifVptZB5yBhkQMwCcGoYMaqginMDHkNSsxDYVnOSNkLPSeN03l1m64hn9zFNyn1gB902+5m3iLPflZUxRitDJcnQ4DlyKIyeQMQZlE2NFTricpeIJ8BhFNX8XI50PbPFNtnGagQEW/6GaKmBquX9eCAReKyWg5c2XarLi7z9kLK4dNpC8kDh2aV/R6MftqXAURXpKQPZniI61ejCAuQz4VM30CkXgGwc8MjymjmcQzpb3oR9Xs841m0Ow0DZZc9uTdy+ZttdOpAu59d5IAQRfearmq2+lcmP0wHystXUIjF9/4gUOvRxL5sqfxrD96fmexeQYKsLL1ItW0AOJzC9322dpYEDumOQZk49vtEEQt0+OwH5qYZem31Y+OJ2nZ3LIJDd9AwaxYTXweX+LvRrZN3s8G3vAFJozDtAK6FkikuJgW1G4mYS3bnY8Z2Il55hSi5+rA3eHBdjK9C5n6dX0AudxR4dakxJm0HpyzOuVvTNmt+PtpM7qEhUyUThCIwCgzKRCJD9zPRUYxShN/xA81Neml6XBXCjt2sumSNL39QKsCUXOtF9hIDpV6WqmUElUzsZwduFlui8bgo4pa9KxYAz8T1D84COsE2c0xm81FOC6nRXKZtpXfgcvRKaylYUL1TnngZZtbrGzzVQpbPEHtZDXHl8CW1Z3+wN8FwGN2U6arQLYtjmO+LLSfpRe58uc1VjpqKM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is very similar to the support that was added to copy_hugetlb_page_range. We simply do a high-granularity walk now, and most of the rest of the code stays the same. Signed-off-by: James Houghton diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 210c6f2b16a5..6c4678b7a07d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5461,16 +5461,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, return ret; } -static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte) +static void move_hugetlb_pte(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, struct hugetlb_pte *src_hpte, + struct hugetlb_pte *dst_hpte) { - struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; spinlock_t *src_ptl, *dst_ptl; pte_t pte; - dst_ptl = huge_pte_lock(h, mm, dst_pte); - src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte); + dst_ptl = hugetlb_pte_lock(dst_hpte); + src_ptl = hugetlb_pte_lockptr(src_hpte); /* * We don't have to worry about the ordering of src and dst ptlocks @@ -5479,8 +5479,8 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, if (src_ptl != dst_ptl) spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); - pte = huge_ptep_get_and_clear(mm, old_addr, src_pte); - set_huge_pte_at(mm, new_addr, dst_pte, pte); + pte = huge_ptep_get_and_clear(mm, old_addr, src_hpte->ptep); + set_huge_pte_at(mm, new_addr, dst_hpte->ptep, pte); if (src_ptl != dst_ptl) spin_unlock(src_ptl); @@ -5498,9 +5498,9 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; unsigned long old_end = old_addr + len; unsigned long last_addr_mask; - pte_t *src_pte, *dst_pte; struct mmu_notifier_range range; bool shared_pmd = false; + struct hugetlb_pte src_hpte, dst_hpte; mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, old_addr, old_end); @@ -5516,28 +5516,35 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, /* Prevent race with file truncation */ hugetlb_vma_lock_write(vma); i_mmap_lock_write(mapping); - for (; old_addr < old_end; old_addr += sz, new_addr += sz) { - src_pte = hugetlb_walk(vma, old_addr, sz); - if (!src_pte) { - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + while (old_addr < old_end) { + if (hugetlb_full_walk(&src_hpte, vma, old_addr)) { + /* The hstate-level PTE wasn't allocated. */ + old_addr = (old_addr | last_addr_mask) + sz; + new_addr = (new_addr | last_addr_mask) + sz; continue; } - if (huge_pte_none(huge_ptep_get(src_pte))) + + if (huge_pte_none(huge_ptep_get(src_hpte.ptep))) { + old_addr += hugetlb_pte_size(&src_hpte); + new_addr += hugetlb_pte_size(&src_hpte); continue; + } - if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) { + if (hugetlb_pte_size(&src_hpte) == sz && + huge_pmd_unshare(mm, vma, old_addr, src_hpte.ptep)) { shared_pmd = true; - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + old_addr = (old_addr | last_addr_mask) + sz; + new_addr = (new_addr | last_addr_mask) + sz; continue; } - dst_pte = huge_pte_alloc(mm, new_vma, new_addr, sz); - if (!dst_pte) + if (hugetlb_full_walk_alloc(&dst_hpte, new_vma, new_addr, + hugetlb_pte_size(&src_hpte))) break; - move_huge_pte(vma, old_addr, new_addr, src_pte, dst_pte); + move_hugetlb_pte(vma, old_addr, new_addr, &src_hpte, &dst_hpte); + old_addr += hugetlb_pte_size(&src_hpte); + new_addr += hugetlb_pte_size(&src_hpte); } if (shared_pmd)