From patchwork Sat Feb 18 00:27:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13145387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7366C636D6 for ; Sat, 18 Feb 2023 00:29:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17404280011; Fri, 17 Feb 2023 19:29:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FD90280002; Fri, 17 Feb 2023 19:29:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E98AD280011; Fri, 17 Feb 2023 19:29:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D0385280002 for ; Fri, 17 Feb 2023 19:29:06 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AE934A019B for ; Sat, 18 Feb 2023 00:29:06 +0000 (UTC) X-FDA: 80478527892.05.91380DF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 01B31180010 for ; Sat, 18 Feb 2023 00:29:04 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jsCsMDMV; spf=pass (imf24.hostedemail.com: domain of 30BvwYwoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=30BvwYwoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676680145; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xFEWCFEVqNzp10q8weB+17bzU0TWKtM8UIz/Xx7DMio=; b=LP2p9Lv0rro9hXhEWp+pwgN/ZgUtZgpeXZ9HKzOViYMssSCy5io8RIb+ateGuGgCbW77T/ jdCUgZj0uyuycDdh8N6Ko3ve8aMWCT/ePZojGwWnz3LncW+okC/eEZJZ+cW2ISK5M4mqoe 3zIQL5fs4IXbChpHPWElQkhDVbZYxZU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jsCsMDMV; spf=pass (imf24.hostedemail.com: domain of 30BvwYwoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=30BvwYwoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676680145; a=rsa-sha256; cv=none; b=8buJul6F3bVY3UEmFb2ms19wqlptiD1eYIz1iTQn1lmqcCxhm+awanXx3QM4M30Atk721r q2ymHi8THamH1NbXfvPIwzbULTaRUuVMXdjFUIuF3sFaxm7c13U5YRNjPZ4Epe3PAgP3ac FGMY6K8y5ke7mWthSSvzMzhZ52M2MCk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5365742524fso24752267b3.13 for ; Fri, 17 Feb 2023 16:29:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xFEWCFEVqNzp10q8weB+17bzU0TWKtM8UIz/Xx7DMio=; b=jsCsMDMV78mjE8aA5hDj4ELvx3b3FYDfUCGtdB3AxH26EGW3FR4Wa7s8y12aCgjJAY xZHdXI0S9zLwyoyC2XXFQ/t878xZ76UJxTZdgaT1614gKQ8S86PPdMjKwYlq97dle3xZ X18JJ5MDR3KedDVpG1uX/f8VVj8pEdf241wOEt1/1ioMygZNzU7w6K5FfESBrPr6rcwL oyCOP/Tffhfxzt33/TtBRunbLaQrs+s+Ng2bav+9PXMaH8T+pd/4F//IKbtKNQ3pXtLn zm3LA/sIhVTcMgTue+SyM1JfkLU6x3PjoLgkHIa6SXkJODW1AhFZZ0jaKNWlzrRQQiMx uOCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xFEWCFEVqNzp10q8weB+17bzU0TWKtM8UIz/Xx7DMio=; b=A58OinjhsaNtbKb6jaE0u8sIS9+k8w6y/5qhxbPK4PL9Du0XpjoRUwbiS5OzKTjsGd wngUMXXBIVoINxYSv4EbYZw7UIZj6i9Qj0MojvjU07rluVJ+tYR82mQ7AYLVWuQl17v2 jO728C8wCTUEIvp2Au24jFg1YYc+eawiOMiPoedcc9ciFSkOp4StTruJa+S4j8ZGz5GC F44+Etxxeo7ZXEJg/K+Ogn4NF9ikEw/XrD06Zw+3W3mefUteafX0LOfrioA0SfBok9wm 8fhtmxyCD9sFyy03YUr80OJCCO5wRkis+pYgVLc+mvyp21PtfhCMKiWI/4gcBsc74cOw X3Lw== X-Gm-Message-State: AO0yUKVIN7Z/DyHxKGZvRSxs8tFcxurHWwCvkFbMoeEs6RphMg4S81aU 7Mn9sTRRA8BDxT3W69zTEbMziRLrhcJlzpxZ X-Google-Smtp-Source: AK7set+l/nyB7A0r6pHwCnr6M8cb+COlb7s5TL7EsXfsO+GgeuyPTV/etrosFrCflWYhmk4cAs1xco6548KS9NkT X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a5b:10e:0:b0:94a:ebba:cba6 with SMTP id 14-20020a5b010e000000b0094aebbacba6mr249759ybx.9.1676680144192; Fri, 17 Feb 2023 16:29:04 -0800 (PST) Date: Sat, 18 Feb 2023 00:27:54 +0000 In-Reply-To: <20230218002819.1486479-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230218002819.1486479-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog Message-ID: <20230218002819.1486479-22-jthoughton@google.com> Subject: [PATCH v2 21/46] hugetlb: add HGM support to hugetlb_follow_page_mask From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu , Andrew Morton Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Frank van der Linden , Jiaqi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 01B31180010 X-Stat-Signature: 9hn8uq9ep5p1bugnr94eku475cotfzbw X-HE-Tag: 1676680144-556709 X-HE-Meta: U2FsdGVkX183bLx69hrRst5VO0One+HTk/arWzZIqYzdNknTXIXu3uOb/lxaQQcXrqbzbHgheIvi3yzIGu+Y0K8CV7hgECFplsxhZOs0iFF4hnbA3vhu3HNJsfWxS1CwIMCAdOKFegsCMM0uyoKqbJBkOWBA26hs/hNMWy+d9OFUGGJaQ0LtzsnTojT1GzFC0bkA69eTSyi47cIeZTH6YYz7QhvqQlq0N3ECdSfunKnr+xcpmHtq3LtskbX+0VrhdVuA7htQaSFDxzORXec4uyQ2SxNeyUMRqVljQme0RKh11zSpfozTKbQ9Gz8TntoqWtgI3dA+24fbQcFLsR00K7+Vgi97Bn4PX+JE3DmoMdK1ImVhCmnGKvxLJPOl2zcbcQ4BDcLIgH7eP6gFghgh645MOjjA0f/tqN9ItyMOH8VlNLi8ynP9OoOPGl0qWcfju8M96fmyrN7UzJOL1F8QU3jVoA4ONZOLNQ2QxyrjMfRIq2mgRcPMAqnVgORlJCXUVAzNEdqT0sOxqzJR8+/QpPvbEqn1g6Sqi3WznU1Nn3baulwuXEzVo/MUfGCGtL85FW20Hns4Nn/nVmALWT0X0rPj4bTXsNddISqGK11fIU20qKxuOFtZBv2JmLtH9lJ1S2t0jVENLXzL2Yj6F85WidbkvJjoGif+ABAt41xKrVajr6DxI8eT9C0g7Ttl+vz7fpVKcFWvWY8T2wmcAhDPNFELTh5OhB9fMCsB8B278ncXr3typhPop16QGvL9trkqjwrjV30SONLdmPpxl4WGVZmZrPMdnV0CoNtuvD8NTtYA8ePStJ4/4Jx2dJJ7Y/kD7O59yPWVu1YgftBhYFjUxW6nozFsQJFL7p42TOjh0A/bguBQyjPIzdZ0BkboYeRw1i+qDt2VfBSvWOAUEWbhcdCtikizSN2VF4tFKjf26vTmVcC0+/aEk2v/x+RbxpFc5LX0C7jBW93zMUIqfwl vbQ/nN7c mCIs+0AxgN0IGqgYKPK0+8GvHIbqKojwZKUU4ovr9BBYBP8Lok06Wgc6JABxobtlHKS04DviD76g1Oebn/XdeEmXSxjXIrZgETpXFtZmEzpuP6Kxdz5wwEafpAzf3vzfe6tlb4raBgbWZfNYzRUTbHl22bPhiJ2lr6aqXO5zRBefhmtHo1nY927LsxrgEHjZ7WmiYGWQ6+IveZF8dt5jszhGadGSdWunUnOpgkAhGuLpGzXWMAfa+APDI0hkkXATFAdGcOxoSprpl5S1d9Z61zdNqrsGBZj+/DT+5p6ouwFt1U/BH3FXyBCujqRTj7FOZ/gu0TBHf2tsc5FFFH+uvm2vgCq/GyV21PrzAU0uHo8UZ2I9vGkZjd6tAPtMgSx7tkUdBNKNfLVUdp18PMmSOuQyScyldm9ciVfS/O56vjzjnwcNMbhC8hEB8d6F8b1T92pXb5xzNe4MdNeYDGAYsoHtTgZic//65KUV7R+7eCLrfYN1Dnr1d1MEpH7nFl/DDZONblHd2YbQoWpDv2xu4NpmSe8g/1EOHzqRGJZPi6gDewXi05TEvM4z4Bd3ctSgqOdNP59rHjHMHyZb3Tg1t7dNgrwvCaPM/LtGxLjr4AV7bmfhcmfY7mxX0dgngxtI7IDfVnA15mU5MPHe2mQNGyxP4n2RyX/kaupJOmaKFN4IwUjcqQRA9OI5867WcnfW7gaWSgX2EgdPndz0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The change here is very simple: do a high-granularity walk. Signed-off-by: James Houghton diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c26b040f4fb5..693332b7e186 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6655,11 +6655,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags) { struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); struct page *page = NULL; spinlock_t *ptl; - pte_t *pte, entry; + pte_t entry; + struct hugetlb_pte hpte; /* * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via @@ -6669,13 +6668,24 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return NULL; hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) + + if (hugetlb_full_walk(&hpte, vma, address)) goto out_unlock; - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); +retry: + ptl = hugetlb_pte_lock(&hpte); + entry = huge_ptep_get(hpte.ptep); if (pte_present(entry)) { + if (unlikely(!hugetlb_pte_present_leaf(&hpte, entry))) { + /* + * We raced with someone splitting from under us. + * Keep walking to get to the real leaf. + */ + spin_unlock(ptl); + hugetlb_full_walk_continue(&hpte, vma, address); + goto retry; + } + page = pte_page(entry) + ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); /*