From patchwork Tue Aug 10 06:26:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12428055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBC17C432BE for ; Tue, 10 Aug 2021 06:26:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 62CFE610A3 for ; Tue, 10 Aug 2021 06:26:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 62CFE610A3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 4389E6B0073; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C35A8D0001; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A1236B0072; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 0FF1A6B0072 for ; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7A3E81D5EA for ; Tue, 10 Aug 2021 06:26:26 +0000 (UTC) X-FDA: 78458186772.25.096A071 Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) by imf15.hostedemail.com (Postfix) with ESMTP id 113C9D004985 for ; Tue, 10 Aug 2021 06:26:25 +0000 (UTC) Received: by mail-lf1-f53.google.com with SMTP id x27so16605406lfu.5 for ; Mon, 09 Aug 2021 23:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=a680rZ8UxFSfjPWaIbbDiJ7ffs2v95N8x5hz5VhhuJc=; b=Kaoo/+qo6x9ifc5IkxRhr2Abo2QlEMA2dumoSO40VGVq7z/mGplRxZh1R6ECCoxoxr u9HREgS/W829rPS0pkS0RfL888/8shlAxvPErro/fnKu0x/ia/fFqVy7I0OR3hbKwnny Mgtbj4FEi5xHpzLLIWjdHYLQMakFwA3MtOz1Q/Q/0z55BstDNwNs3Sr0PxbkfQ74s26n EMaZwexRYdulPeGaXnTt/rzGOxYskklZsEcIWKJwg6O1zB9wdulpPqK4V0GJ5tYy5gKx F/+YZ0nnZED0I+ErMQPOmGNBF6FHR3quWI2MbUjl2xj8hpIu6XLoDUbn1IcH82zQ4GY6 PvMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a680rZ8UxFSfjPWaIbbDiJ7ffs2v95N8x5hz5VhhuJc=; b=QowgXdjnd+QJzI6ffSJYgybKT9yXQIxcRl9F5D43Ruyjg6zKRRDCLxgb3QY/+I2bR5 DUrrROZ0BwuUPZTsv+Jn7wVY/XwslW5N4Ne4IJhIWQTmMgqdPO7igFkfbxzKCm7RDtxR LZl4B7h7qsLbkRansHTjUG3wGl/GQe513wXgt5oCbbqdTl4Xd4xg5tdrY+1uzWQs2Q9J g0DZEebK9DBsSeXsIGLAYOKMO6liSWHJmpG6ZAvX7R4CFDZ+hTguCxc5cgcbpJe3GWwp +JnueT0bG/apXScwFCcTGeYKSOyxgT5n9+HKw46ZfOz0CyEPxECYRUc+uJlSExOIdps2 cijg== X-Gm-Message-State: AOAM533zD1q/dS+3eaPE4KkiTHJdDhZV288NUntdkhFvtmTaibpcVF2R TyQTFfl1fVH2thaRlTyt0CS5jA== X-Google-Smtp-Source: ABdhPJyEYmaNwfXNRJDS0I7KH549XMeGo9eFzucZFhdxX1QsjktVT8E12hyntGfDpzdtgWo86ZP2nQ== X-Received: by 2002:a05:6512:2626:: with SMTP id bt38mr4712647lfb.181.1628576784615; Mon, 09 Aug 2021 23:26:24 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id n2sm42195lfl.36.2021.08.09.23.26.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 23:26:22 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 9224C1028F5; Tue, 10 Aug 2021 09:26:34 +0300 (+03) To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 1/5] mm: Add support for unaccepted memory Date: Tue, 10 Aug 2021 09:26:22 +0300 Message-Id: <20210810062626.1012-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> References: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 113C9D004985 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=shutemov-name.20150623.gappssmtp.com header.s=20150623 header.b="Kaoo/+qo"; dmarc=none; spf=none (imf15.hostedemail.com: domain of kirill@shutemov.name has no SPF policy when checking 209.85.167.53) smtp.mailfrom=kirill@shutemov.name X-Stat-Signature: wweugkfkeetr5njgncbweemb86pz9r59 X-HE-Tag: 1628576785-502022 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: UEFI Specification version 2.9 introduces concept of memory acceptance: Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtrual Machine platform. Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptation until memory is needed. It lowers boot time and reduces memory overhead. Support of such memory requires few changes in core-mm code: - memblock has to accept memory on allocation; - page allocator has to accept memory on the first allocation of the page; Memblock change is trivial. Page allocator is modified to accept pages on the first allocation. PageOffline() is used to indicate that the page requires acceptance. The flag currently used by hotplug and balloon. Such pages are not available to page allocator. An architecture has to provide three helpers if it wants to support unaccepted memory: - accept_memory() makes a range of physical addresses accepted. - maybe_set_page_offline() marks a page PageOffline() if it requires acceptance. Used during boot to put pages on free lists. - clear_page_offline() clears makes a page accepted and clears PageOffline(). Signed-off-by: Kirill A. Shutemov --- mm/internal.h | 14 ++++++++++++++ mm/memblock.c | 1 + mm/page_alloc.c | 13 ++++++++++++- 3 files changed, 27 insertions(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index 31ff935b2547..d2fc8a17fbe0 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -662,4 +662,18 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, unsigned long addr, int page_nid, int *flags); +#ifndef CONFIG_UNACCEPTED_MEMORY +static inline void maybe_set_page_offline(struct page *page, unsigned int order) +{ +} + +static inline void clear_page_offline(struct page *page, unsigned int order) +{ +} + +static inline void accept_memory(phys_addr_t start, phys_addr_t end) +{ +} +#endif + #endif /* __MM_INTERNAL_H */ diff --git a/mm/memblock.c b/mm/memblock.c index 28a813d9e955..8c1bf08f2b0b 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -1370,6 +1370,7 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, */ kmemleak_alloc_phys(found, size, 0, 0); + accept_memory(found, found + size); return found; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 856b175c15a4..892347d9a507 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -899,6 +899,9 @@ static inline bool page_is_buddy(struct page *page, struct page *buddy, if (buddy_order(buddy) != order) return false; + if (PageOffline(buddy) || PageOffline(page)) + return false; + /* * zone check is done late to avoid uselessly calculating * zone/node ids for pages that could never merge. @@ -1001,6 +1004,9 @@ static inline void del_page_from_free_list(struct page *page, struct zone *zone, if (page_reported(page)) __ClearPageReported(page); + if (PageOffline(page)) + clear_page_offline(page, order); + list_del(&page->lru); __ClearPageBuddy(page); set_page_private(page, 0); @@ -1165,7 +1171,8 @@ static inline void __free_one_page(struct page *page, static inline bool page_expected_state(struct page *page, unsigned long check_flags) { - if (unlikely(atomic_read(&page->_mapcount) != -1)) + if (unlikely(atomic_read(&page->_mapcount) != -1) && + !PageOffline(page)) return false; if (unlikely((unsigned long)page->mapping | @@ -1748,6 +1755,8 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn, { if (early_page_uninitialised(pfn)) return; + + maybe_set_page_offline(page, order); __free_pages_core(page, order); } @@ -1839,10 +1848,12 @@ static void __init deferred_free_range(unsigned long pfn, if (nr_pages == pageblock_nr_pages && (pfn & (pageblock_nr_pages - 1)) == 0) { set_pageblock_migratetype(page, MIGRATE_MOVABLE); + maybe_set_page_offline(page, pageblock_order); __free_pages_core(page, pageblock_order); return; } + accept_memory(pfn << PAGE_SHIFT, (pfn + nr_pages) << PAGE_SHIFT); for (i = 0; i < nr_pages; i++, page++, pfn++) { if ((pfn & (pageblock_nr_pages - 1)) == 0) set_pageblock_migratetype(page, MIGRATE_MOVABLE); From patchwork Tue Aug 10 06:26:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12428061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 617AEC43214 for ; Tue, 10 Aug 2021 06:26:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D8D1861077 for ; Tue, 10 Aug 2021 06:26:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D8D1861077 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E86756B0075; Tue, 10 Aug 2021 02:26:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E22DF8D0002; Tue, 10 Aug 2021 02:26:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACC4D6B0078; Tue, 10 Aug 2021 02:26:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 939EE6B0074 for ; Tue, 10 Aug 2021 02:26:28 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 426928249980 for ; Tue, 10 Aug 2021 06:26:28 +0000 (UTC) X-FDA: 78458186856.04.5E5CB17 Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by imf02.hostedemail.com (Postfix) with ESMTP id F21537006203 for ; Tue, 10 Aug 2021 06:26:27 +0000 (UTC) Received: by mail-lj1-f176.google.com with SMTP id u13so27400732lje.5 for ; Mon, 09 Aug 2021 23:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wx+RYAAVAPw18oVnn3qZR4sMu11WM9+OwnA33lJi4Gw=; b=KjNnlspysltiYL9VG1uTtzPa26xJF/4zf28T9Q8NUyea39Hie4Cg6Yt7e1mDZ6tmjH Zuj6D2wx1SgT7ihXTirifdxPF6O/BsvSUxiZhzselypHb9i167AS9inRV7TI+juMqt/a XlGmRsV3i78lbRQndTrBHd7ScFISM+QkYxg4D037nVJVBZ1xewPnW2hzvGdMWfh7Qsss hO2Cxqlt7TjnP7G0OHspAPWVJUoapBNUcPFF8KKoYTTJW832BS9lr/detTiQzwi00+6D a+lTZR5WOrVRmzhEOA2eTA/bawQGWKtnkAgaRDpYhdGakucGDyC2OphNzutuJ3BGFWoz rVYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wx+RYAAVAPw18oVnn3qZR4sMu11WM9+OwnA33lJi4Gw=; b=DDopvtgQWbU0VRHb/8OWNO+0W6FumrpPgJiUYTa4N3xThCsWxH3qBZTgxMCGgp33O/ YzLtnmWK+TD7lqycg/Bt7KCBxyB5sy3R7pLNhD5VSmLJr49BbMQBLqeygNoVzk4OBXIm loNoocFnvzg1OGwp+NTGxnqHOKVuaAeMafRwboaeZj/jFwx6713p7nYtJCp96hZpXIkG ZGt2sIfXZODsYTuGZp9jFKB/+PccyewBcpW5n53UIP72CzfIOMyhiGfC2ILwJ4E7JEIy mXXqoA3Erw7dqLUB+cINQYkIRH4DWEvCFLVD1sZYcb6TEKOOZSPAxxAjhAv7Js6UMlN+ fKMA== X-Gm-Message-State: AOAM531jeV5aaLXVA1FvueTdwKVCCtZ1SK6pAGGZt8hKTBf0/fyee8JC cTRhBuJrpm81r64kXnZWMOO5wg== X-Google-Smtp-Source: ABdhPJwWBQ+K3JlUHh1r9KuMzK5f88CTEesYRXokmMYpujHyCswZFi39bgh9hPbmyaRICi3KPxpsAA== X-Received: by 2002:a2e:a4c9:: with SMTP id p9mr18511425ljm.437.1628576786272; Mon, 09 Aug 2021 23:26:26 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id z24sm1948290lfr.105.2021.08.09.23.26.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 23:26:22 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 9C30E102BD9; Tue, 10 Aug 2021 09:26:34 +0300 (+03) To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 2/5] efi/x86: Implement support for unaccepted memory Date: Tue, 10 Aug 2021 09:26:23 +0300 Message-Id: <20210810062626.1012-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> References: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: F21537006203 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=shutemov-name.20150623.gappssmtp.com header.s=20150623 header.b=KjNnlspy; dmarc=none; spf=none (imf02.hostedemail.com: domain of kirill@shutemov.name has no SPF policy when checking 209.85.208.176) smtp.mailfrom=kirill@shutemov.name X-Stat-Signature: 1ms6mpt6gu1jbmw83zyn4ffta7jqtb74 X-HE-Tag: 1628576787-765785 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: UEFI Specification version 2.9 introduces concept of memory acceptance: Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtrual Machine platform. Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptation until memory is needed. It lowers boot time and reduces memory overhead. Kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory. Range based tracking works fine for firmware, but it gets bulky for kernel: e820 has to be modified on every page acceptance. It leads to table fragmentation, but there's a limited number of entries in the e820 table Other option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents 2MiB in the address space: one 4k page is enough to track 64GiB or physical address space. In the worst case scenario -- a huge hole in the middle of the address space -- we would need 256MiB to handle 4PiB of the address space. Any unaccepted memory that is not aligned to 2M get accepted upfront. The bitmap allocated and constructed in EFI stub and passed down to kernel via boot_params. allocate_e820() allocates the bitmap if unaccepted memory present according to the maximum address in the memory map. The same boot_params.unaccepted_memory can be used to pass the bitmap between two kernel on kexec, but the use-case is not yet implemented. Signed-off-by: Kirill A. Shutemov --- Documentation/x86/zero-page.rst | 1 + arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/bitmap.c | 24 +++++++ arch/x86/boot/compressed/unaccepted_memory.c | 36 ++++++++++ arch/x86/include/asm/unaccepted_memory.h | 12 ++++ arch/x86/include/uapi/asm/bootparam.h | 3 +- drivers/firmware/efi/Kconfig | 12 ++++ drivers/firmware/efi/efi.c | 1 + drivers/firmware/efi/libstub/x86-stub.c | 75 ++++++++++++++++---- include/linux/efi.h | 3 +- 10 files changed, 153 insertions(+), 15 deletions(-) create mode 100644 arch/x86/boot/compressed/bitmap.c create mode 100644 arch/x86/boot/compressed/unaccepted_memory.c create mode 100644 arch/x86/include/asm/unaccepted_memory.h diff --git a/Documentation/x86/zero-page.rst b/Documentation/x86/zero-page.rst index f088f5881666..8e3447a4b373 100644 --- a/Documentation/x86/zero-page.rst +++ b/Documentation/x86/zero-page.rst @@ -42,4 +42,5 @@ Offset/Size Proto Name Meaning 2D0/A00 ALL e820_table E820 memory map table (array of struct e820_entry) D00/1EC ALL eddbuf EDD data (array of struct edd_info) +ECC/008 ALL unaccepted_memory Bitmap of unaccepted memory (1bit == 2M) =========== ===== ======================= ================================================= diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 1bfe30ebadbe..f5b49e74d728 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -100,6 +100,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/bitmap.o $(obj)/unaccepted_memory.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_thunk_$(BITS).o efi-obj-$(CONFIG_EFI_STUB) = $(objtree)/drivers/firmware/efi/libstub/lib.a diff --git a/arch/x86/boot/compressed/bitmap.c b/arch/x86/boot/compressed/bitmap.c new file mode 100644 index 000000000000..bf58b259380a --- /dev/null +++ b/arch/x86/boot/compressed/bitmap.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Taken from lib/string.c */ + +#include + +void __bitmap_set(unsigned long *map, unsigned int start, int len) +{ + unsigned long *p = map + BIT_WORD(start); + const unsigned int size = start + len; + int bits_to_set = BITS_PER_LONG - (start % BITS_PER_LONG); + unsigned long mask_to_set = BITMAP_FIRST_WORD_MASK(start); + + while (len - bits_to_set >= 0) { + *p |= mask_to_set; + len -= bits_to_set; + bits_to_set = BITS_PER_LONG; + mask_to_set = ~0UL; + p++; + } + if (len) { + mask_to_set &= BITMAP_LAST_WORD_MASK(size); + *p |= mask_to_set; + } +} diff --git a/arch/x86/boot/compressed/unaccepted_memory.c b/arch/x86/boot/compressed/unaccepted_memory.c new file mode 100644 index 000000000000..c2eca85b5073 --- /dev/null +++ b/arch/x86/boot/compressed/unaccepted_memory.c @@ -0,0 +1,36 @@ +#include "error.h" +#include "misc.h" + +static inline void __accept_memory(phys_addr_t start, phys_addr_t end) +{ + /* Platform-specific memory-acceptance call goes here */ + error("Cannot accept memory"); +} + +void mark_unaccepted(struct boot_params *params, u64 start, u64 num) +{ + u64 end = start + num * PAGE_SIZE; + unsigned int npages; + + if ((start & PMD_MASK) == (end & PMD_MASK)) { + npages = (end - start) / PAGE_SIZE; + __accept_memory(start, start + npages * PAGE_SIZE); + return; + } + + if (start & ~PMD_MASK) { + npages = (round_up(start, PMD_SIZE) - start) / PAGE_SIZE; + __accept_memory(start, start + npages * PAGE_SIZE); + start = round_up(start, PMD_SIZE); + } + + if (end & ~PMD_MASK) { + npages = (end - round_down(end, PMD_SIZE)) / PAGE_SIZE; + end = round_down(end, PMD_SIZE); + __accept_memory(end, end + npages * PAGE_SIZE); + } + + npages = (end - start) / PMD_SIZE; + bitmap_set((unsigned long *)params->unaccepted_memory, + start / PMD_SIZE, npages); +} diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h new file mode 100644 index 000000000000..cbc24040b853 --- /dev/null +++ b/arch/x86/include/asm/unaccepted_memory.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2020 Intel Corporation */ +#ifndef _ASM_X86_UNACCEPTED_MEMORY_H +#define _ASM_X86_UNACCEPTED_MEMORY_H + +#include + +struct boot_params; + +void mark_unaccepted(struct boot_params *params, u64 start, u64 num); + +#endif diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h index b25d3f82c2f3..16bc686a198d 100644 --- a/arch/x86/include/uapi/asm/bootparam.h +++ b/arch/x86/include/uapi/asm/bootparam.h @@ -217,7 +217,8 @@ struct boot_params { struct boot_e820_entry e820_table[E820_MAX_ENTRIES_ZEROPAGE]; /* 0x2d0 */ __u8 _pad8[48]; /* 0xcd0 */ struct edd_info eddbuf[EDDMAXNR]; /* 0xd00 */ - __u8 _pad9[276]; /* 0xeec */ + __u64 unaccepted_memory; /* 0xeec */ + __u8 _pad9[268]; /* 0xef4 */ } __attribute__((packed)); /** diff --git a/drivers/firmware/efi/Kconfig b/drivers/firmware/efi/Kconfig index 2c3dac5ecb36..e13b584cdd80 100644 --- a/drivers/firmware/efi/Kconfig +++ b/drivers/firmware/efi/Kconfig @@ -243,6 +243,18 @@ config EFI_DISABLE_PCI_DMA options "efi=disable_early_pci_dma" or "efi=no_disable_early_pci_dma" may be used to override this option. +config UNACCEPTED_MEMORY + bool + depends on EFI_STUB + help + Some Virtual Machine platforms, such as Intel TDX, introduce + the concept of memory acceptance, requiring memory to be accepted + before it can be used by the guest. This protects against a class of + attacks by the virtual machine platform. + + This option adds support for unaccepted memory and makes such memory + usable by kernel. + endmenu config EFI_EMBEDDED_FIRMWARE diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index 847f33ffc4ae..c6b8a1c5a87f 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -737,6 +737,7 @@ static __initdata char memory_type_name[][13] = { "MMIO Port", "PAL Code", "Persistent", + "Unaccepted", }; char * __init efi_md_typeattr_format(char *buf, size_t size, diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index f14c4ff5839f..e67ec1245f10 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -9,12 +9,14 @@ #include #include #include +#include #include #include #include #include #include +#include #include "efistub.h" @@ -504,6 +506,12 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s e820_type = E820_TYPE_PMEM; break; + case EFI_UNACCEPTED_MEMORY: + if (!IS_ENABLED(CONFIG_UNACCEPTED_MEMORY)) + continue; + e820_type = E820_TYPE_RAM; + mark_unaccepted(params, d->phys_addr, d->num_pages); + break; default: continue; } @@ -569,30 +577,71 @@ static efi_status_t alloc_e820ext(u32 nr_desc, struct setup_data **e820ext, } static efi_status_t allocate_e820(struct boot_params *params, + struct efi_boot_memmap *map, struct setup_data **e820ext, u32 *e820ext_size) { - unsigned long map_size, desc_size, map_key; efi_status_t status; - __u32 nr_desc, desc_version; - - /* Only need the size of the mem map and size of each mem descriptor */ - map_size = 0; - status = efi_bs_call(get_memory_map, &map_size, NULL, &map_key, - &desc_size, &desc_version); - if (status != EFI_BUFFER_TOO_SMALL) - return (status != EFI_SUCCESS) ? status : EFI_UNSUPPORTED; + __u32 nr_desc; + bool unaccepted_memory_present = false; + u64 max_addr = 0; + int i; - nr_desc = map_size / desc_size + EFI_MMAP_NR_SLACK_SLOTS; + status = efi_get_memory_map(map); + if (status != EFI_SUCCESS) + return status; - if (nr_desc > ARRAY_SIZE(params->e820_table)) { - u32 nr_e820ext = nr_desc - ARRAY_SIZE(params->e820_table); + nr_desc = *map->map_size / *map->desc_size; + if (nr_desc > ARRAY_SIZE(params->e820_table) - EFI_MMAP_NR_SLACK_SLOTS) { + u32 nr_e820ext = nr_desc - ARRAY_SIZE(params->e820_table) - + EFI_MMAP_NR_SLACK_SLOTS; status = alloc_e820ext(nr_e820ext, e820ext, e820ext_size); if (status != EFI_SUCCESS) return status; } + if (!IS_ENABLED(CONFIG_UNACCEPTED_MEMORY)) + return EFI_SUCCESS; + + /* Check if there's any unaccepted memory and find the max address */ + for (i = 0; i < nr_desc; i++) { + efi_memory_desc_t *d; + + d = efi_early_memdesc_ptr(*map->map, *map->desc_size, i); + if (d->type == EFI_UNACCEPTED_MEMORY) + unaccepted_memory_present = true; + if (d->phys_addr + d->num_pages * PAGE_SIZE > max_addr) + max_addr = d->phys_addr + d->num_pages * PAGE_SIZE; + } + + /* + * If unaccepted memory present allocate a bitmap to track what memory + * has to be accepted before access. + * + * One bit in the bitmap represents 2MiB in the address space: one 4k + * page is enough to track 64GiB or physical address space. + * + * In the worst case scenario -- a huge hole in the middle of the + * address space -- we would need 256MiB to handle 4PiB of the address + * space. + * + * TODO: handle situation if params->unaccepted_memory has already set. + * It's required to deal with kexec. + */ + if (unaccepted_memory_present) { + unsigned long *unaccepted_memory = NULL; + u64 size = DIV_ROUND_UP(max_addr, PMD_SIZE * BITS_PER_BYTE); + + status = efi_allocate_pages(size, + (unsigned long *)&unaccepted_memory, + ULONG_MAX); + if (status != EFI_SUCCESS) + return status; + memset(unaccepted_memory, 0, size); + params->unaccepted_memory = (u64)unaccepted_memory; + } + return EFI_SUCCESS; } @@ -642,7 +691,7 @@ static efi_status_t exit_boot(struct boot_params *boot_params, void *handle) priv.boot_params = boot_params; priv.efi = &boot_params->efi_info; - status = allocate_e820(boot_params, &e820ext, &e820ext_size); + status = allocate_e820(boot_params, &map, &e820ext, &e820ext_size); if (status != EFI_SUCCESS) return status; diff --git a/include/linux/efi.h b/include/linux/efi.h index 6b5d36babfcc..d43cc872b582 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -108,7 +108,8 @@ typedef struct { #define EFI_MEMORY_MAPPED_IO_PORT_SPACE 12 #define EFI_PAL_CODE 13 #define EFI_PERSISTENT_MEMORY 14 -#define EFI_MAX_MEMORY_TYPE 15 +#define EFI_UNACCEPTED_MEMORY 15 +#define EFI_MAX_MEMORY_TYPE 16 /* Attribute values: */ #define EFI_MEMORY_UC ((u64)0x0000000000000001ULL) /* uncached */ From patchwork Tue Aug 10 06:26:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12428059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3B68C4320A for ; Tue, 10 Aug 2021 06:26:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 742F36105A for ; Tue, 10 Aug 2021 06:26:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 742F36105A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 9DC918D0001; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F35A6B0075; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 599348D0002; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id 25F736B0075 for ; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id AB96B8249980 for ; Tue, 10 Aug 2021 06:26:26 +0000 (UTC) X-FDA: 78458186772.31.D2F4346 Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) by imf08.hostedemail.com (Postfix) with ESMTP id 6B24F3000246 for ; Tue, 10 Aug 2021 06:26:26 +0000 (UTC) Received: by mail-lf1-f49.google.com with SMTP id w20so12645481lfu.7 for ; Mon, 09 Aug 2021 23:26:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yhM9rzQZh1hFlmIsdV4JGrfhMhv28jDR+Y6mMaIyOd4=; b=iI6obK6IDsJ5WRxV6OM1M/tAx2RUXXdpWjOOljATFp3gKmjcJd16cMca3CRczpVk90 vdUUedOazrl3bkG2F0IfOJ6AItLxNYJwCF1n4JKF50+TflcgrRKXqnKIm/J/THTyieZr KsM8HcwID/NK5sqFV1M5hQPxhiER2JGcI0YABCRIa4iVTkIyyqaNxusuU50potUPb3Oi PUB+8f9Ij8rl+y/qgnVCYh7HdM2F7I53V4Dh1DLgERtHiB2qPWoUfEd0GCbBsUCX6DAg /nKOFs0c2RkzNtTU4ew3Npu0B4ABZRqH7MBpxW8V+hsglvKIqewABkD2rTgSDih6C3Up n1uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yhM9rzQZh1hFlmIsdV4JGrfhMhv28jDR+Y6mMaIyOd4=; b=UpnSc/mtqPScd17iniBdCVm4dmso8KCNo2zxhlVclmvTiaWXEssCpZ2kuyy2VDwTzt LmsvQOW72DCEBsGvJBFJCI3UEhCLpLBYGBD8po5ysTPzlccJWA1kJbLKujrMABMeN0HZ YSB4+iUNhdXvXVKYhu8rmhJvhg5BLSnxqTIYQ+FJ/vr+ZuhMdh7TCSiV8jhdl8BLy3la e4rXSTmbUrVl+OdPiZsaIzvdvYrFlyQ1C+NPogXQCYLVKLkeIk57GKpzGV1WUfn4E1KH FclP3bhmn391LBQW/k/Qf5tOM0plppIx/f44vGpfsALSzYhXctOikVROb/p3Mv9BqUQT gIwQ== X-Gm-Message-State: AOAM532mEHDXEIsc3e6azeAfP1nToY/G/EasHH3iUOFdAXuqKceKd7jz YbOn3I96hxsBNPS5cio9MNWSYw== X-Google-Smtp-Source: ABdhPJywOJGNlI7IqnnlU0lsO+YPTdIUQE6GK58teoRywSZHJeH0Ifngw9u6S0AYbWyYjWXtNj2VvA== X-Received: by 2002:a05:6512:e86:: with SMTP id bi6mr21473291lfb.592.1628576784944; Mon, 09 Aug 2021 23:26:24 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id w27sm264085lfa.308.2021.08.09.23.26.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 23:26:22 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id A60D9102BDA; Tue, 10 Aug 2021 09:26:34 +0300 (+03) To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 3/5] x86/boot/compressed: Handle unaccepted memory Date: Tue, 10 Aug 2021 09:26:24 +0300 Message-Id: <20210810062626.1012-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> References: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6B24F3000246 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=shutemov-name.20150623.gappssmtp.com header.s=20150623 header.b=iI6obK6I; dmarc=none; spf=none (imf08.hostedemail.com: domain of kirill@shutemov.name has no SPF policy when checking 209.85.167.49) smtp.mailfrom=kirill@shutemov.name X-Stat-Signature: ic85sc6bznowf89z5ieq4aixdtuc3it7 X-HE-Tag: 1628576786-106350 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firmware is responsible for accepting memory where compressed kernel image and initrd land. But kernel has to accept memory for decompression buffer: accept memory just before decompression starts KASLR allowed to use unaccepted memory for output buffer. Signed-off-by: Kirill A. Shutemov --- arch/x86/boot/compressed/bitmap.c | 62 ++++++++++++++++++++ arch/x86/boot/compressed/kaslr.c | 14 ++++- arch/x86/boot/compressed/misc.c | 9 +++ arch/x86/boot/compressed/unaccepted_memory.c | 13 ++++ arch/x86/include/asm/unaccepted_memory.h | 2 + 5 files changed, 98 insertions(+), 2 deletions(-) diff --git a/arch/x86/boot/compressed/bitmap.c b/arch/x86/boot/compressed/bitmap.c index bf58b259380a..ba2de61c0823 100644 --- a/arch/x86/boot/compressed/bitmap.c +++ b/arch/x86/boot/compressed/bitmap.c @@ -2,6 +2,48 @@ /* Taken from lib/string.c */ #include +#include +#include + +unsigned long _find_next_bit(const unsigned long *addr1, + const unsigned long *addr2, unsigned long nbits, + unsigned long start, unsigned long invert, unsigned long le) +{ + unsigned long tmp, mask; + + if (unlikely(start >= nbits)) + return nbits; + + tmp = addr1[start / BITS_PER_LONG]; + if (addr2) + tmp &= addr2[start / BITS_PER_LONG]; + tmp ^= invert; + + /* Handle 1st word. */ + mask = BITMAP_FIRST_WORD_MASK(start); + if (le) + mask = swab(mask); + + tmp &= mask; + + start = round_down(start, BITS_PER_LONG); + + while (!tmp) { + start += BITS_PER_LONG; + if (start >= nbits) + return nbits; + + tmp = addr1[start / BITS_PER_LONG]; + if (addr2) + tmp &= addr2[start / BITS_PER_LONG]; + tmp ^= invert; + } + + if (le) + tmp = swab(tmp); + + return min(start + __ffs(tmp), nbits); +} void __bitmap_set(unsigned long *map, unsigned int start, int len) { @@ -22,3 +64,23 @@ void __bitmap_set(unsigned long *map, unsigned int start, int len) *p |= mask_to_set; } } + +void __bitmap_clear(unsigned long *map, unsigned int start, int len) +{ + unsigned long *p = map + BIT_WORD(start); + const unsigned int size = start + len; + int bits_to_clear = BITS_PER_LONG - (start % BITS_PER_LONG); + unsigned long mask_to_clear = BITMAP_FIRST_WORD_MASK(start); + + while (len - bits_to_clear >= 0) { + *p &= ~mask_to_clear; + len -= bits_to_clear; + bits_to_clear = BITS_PER_LONG; + mask_to_clear = ~0UL; + p++; + } + if (len) { + mask_to_clear &= BITMAP_LAST_WORD_MASK(size); + *p &= ~mask_to_clear; + } +} diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c index e36690778497..89cdc17dcbf1 100644 --- a/arch/x86/boot/compressed/kaslr.c +++ b/arch/x86/boot/compressed/kaslr.c @@ -729,10 +729,20 @@ process_efi_entries(unsigned long minimum, unsigned long image_size) * but in practice there's firmware where using that memory leads * to crashes. * - * Only EFI_CONVENTIONAL_MEMORY is guaranteed to be free. + * Only EFI_CONVENTIONAL_MEMORY and EFI_UNACCEPTED_MEMORY (if + * supported) are guaranteed to be free. */ - if (md->type != EFI_CONVENTIONAL_MEMORY) + + switch (md->type) { + case EFI_CONVENTIONAL_MEMORY: + break; + case EFI_UNACCEPTED_MEMORY: + if (IS_ENABLED(CONFIG_UNACCEPTED_MEMORY)) + break; continue; + default: + continue; + } if (efi_soft_reserve_enabled() && (md->attribute & EFI_MEMORY_SP)) diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index 743f13ea25c1..eeefcde8394d 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -18,6 +18,7 @@ #include "../string.h" #include "../voffset.h" #include +#include /* * WARNING!! @@ -435,6 +436,14 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap, #endif debug_putstr("\nDecompressing Linux... "); + + if (IS_ENABLED(CONFIG_UNACCEPTED_MEMORY) && + boot_params->unaccepted_memory) { + debug_putstr("Accepting memory... "); + accept_memory((phys_addr_t)output, + (phys_addr_t)output + needed_size); + } + __decompress(input_data, input_len, NULL, NULL, output, output_len, NULL, error); parse_elf(output); diff --git a/arch/x86/boot/compressed/unaccepted_memory.c b/arch/x86/boot/compressed/unaccepted_memory.c index c2eca85b5073..17b70627b0cd 100644 --- a/arch/x86/boot/compressed/unaccepted_memory.c +++ b/arch/x86/boot/compressed/unaccepted_memory.c @@ -34,3 +34,16 @@ void mark_unaccepted(struct boot_params *params, u64 start, u64 num) bitmap_set((unsigned long *)params->unaccepted_memory, start / PMD_SIZE, npages); } + +void accept_memory(phys_addr_t start, phys_addr_t end) +{ + unsigned long *unaccepted_memory; + unsigned int rs, re; + + unaccepted_memory = (unsigned long *)boot_params->unaccepted_memory; + bitmap_for_each_set_region(unaccepted_memory, rs, re, + start / PMD_SIZE, end / PMD_SIZE) { + __accept_memory(rs * PMD_SIZE, re * PMD_SIZE); + bitmap_clear(unaccepted_memory, rs, re - rs); + } +} diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h index cbc24040b853..f1f835d3cd78 100644 --- a/arch/x86/include/asm/unaccepted_memory.h +++ b/arch/x86/include/asm/unaccepted_memory.h @@ -9,4 +9,6 @@ struct boot_params; void mark_unaccepted(struct boot_params *params, u64 start, u64 num); +void accept_memory(phys_addr_t start, phys_addr_t end); + #endif From patchwork Tue Aug 10 06:26:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12428057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2D1FC4338F for ; Tue, 10 Aug 2021 06:26:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7671C6105A for ; Tue, 10 Aug 2021 06:26:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7671C6105A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 7BCBE6B0072; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7470D8D0001; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 46E186B0078; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0184.hostedemail.com [216.40.44.184]) by kanga.kvack.org (Postfix) with ESMTP id 245226B0074 for ; Tue, 10 Aug 2021 02:26:27 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C9331181EC4A2 for ; Tue, 10 Aug 2021 06:26:26 +0000 (UTC) X-FDA: 78458186772.22.2098E1B Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by imf15.hostedemail.com (Postfix) with ESMTP id 8A1CBD004985 for ; Tue, 10 Aug 2021 06:26:26 +0000 (UTC) Received: by mail-lf1-f43.google.com with SMTP id d4so10385962lfk.9 for ; Mon, 09 Aug 2021 23:26:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wBfWTIHuoUkiwmizGdYmrNr5TmKY7E7Jf9eiL0J7uVk=; b=0h4kwTTU9bgsr5+7M2dbKNl6HdsvupLgIPjYSGLs6f8fz3APWo6zJSPPWqNXW4N/Li UunV42SnRXg/QfbOe6YhRwV3FHVUGTQLs/9luwBZqG1bEgUsxo7c8tHkZ+XQFDIuTpU/ dE60LpbSxBQE68n6v6KGu9eg0zHbWg3jw9ysv+LSIJuLieR4bO4yps2uMY/2h7Q1vR2P q/QdcwI+rjdhzncpWhT1oKNYF68WnvwozDBbb0/dok1B+62Pzyuw46SN/Dp7slhPoMUW fd/rPnR6ZCWvPlxEXkK6rmZqS/R7ooKaJrHFi/MtCXKJLO9zPFiKSaryFrn4KeSfJFd0 VQ5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wBfWTIHuoUkiwmizGdYmrNr5TmKY7E7Jf9eiL0J7uVk=; b=psEcionckYcKVdRXV41LQwuj1AU4ds5whAABNlya7d5u8sg7ECKUsR5WAh3zWGgI09 0XqgEluB2Ek+/aG3/fUkBsqJGcutG+zvMxorZE0Bi17OHtfwT/HB6JbfK9DR8CSXMDDi w8FHtr5vcThRcfmA1iaqxi7CgOiIsuyWAREGX7GSr+9SSzxxVRe3FddhfT6PmAEWXXBj KQSECbmfZ+f3AbnaM9rgGAGO+B1OL7olSVO7Lk9UEgc8fjz9dF88LKOEPVwsXOHkjxY8 8EyCHf8Ik9HLwQYaQvf2r+Boax1CVv5e/QAf8kjlvmbB6crmUxjIh6W0aToPaFoxCe15 x4Mg== X-Gm-Message-State: AOAM533yB0vbzF/tCs1VMwpLCD8agzSjDtiO088mah7gnO1+6nH9LoyR 2hPM7OcedIR1KIvjzgZO1O6trw== X-Google-Smtp-Source: ABdhPJwRONZllplmQufLj5RnClnVIfE2Nmqatiq9pV28+ucprKnq9Or8NIypwNI4s9sGELeie9B4GA== X-Received: by 2002:ac2:443a:: with SMTP id w26mr3042148lfl.537.1628576785236; Mon, 09 Aug 2021 23:26:25 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id m13sm1948755lfu.5.2021.08.09.23.26.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 23:26:22 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id AECA5102BDB; Tue, 10 Aug 2021 09:26:34 +0300 (+03) To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 4/5] x86/mm: Provide helpers for unaccepted memory Date: Tue, 10 Aug 2021 09:26:25 +0300 Message-Id: <20210810062626.1012-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> References: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=shutemov-name.20150623.gappssmtp.com header.s=20150623 header.b=0h4kwTTU; spf=none (imf15.hostedemail.com: domain of kirill@shutemov.name has no SPF policy when checking 209.85.167.43) smtp.mailfrom=kirill@shutemov.name; dmarc=none X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8A1CBD004985 X-Stat-Signature: t331wxdohz3eony497za9jceqpy6dpii X-HE-Tag: 1628576786-965893 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Core-mm requires few helpers to support unaccepted memory: - accept_memory() checks the range of addresses against the bitmap and accept memory if needed; - maybe_set_page_offline() checks the bitmap and marks a page with PageOffline() if memory acceptance is required on the first allocation of the page. - clear_page_offline() accepts memory for the page and clears PageOffline(). Signed-off-by: Kirill A. Shutemov --- arch/x86/boot/compressed/unaccepted_memory.c | 3 +- arch/x86/include/asm/page.h | 5 ++ arch/x86/include/asm/unaccepted_memory.h | 3 + arch/x86/mm/Makefile | 2 + arch/x86/mm/unaccepted_memory.c | 80 ++++++++++++++++++++ 5 files changed, 91 insertions(+), 2 deletions(-) create mode 100644 arch/x86/mm/unaccepted_memory.c diff --git a/arch/x86/boot/compressed/unaccepted_memory.c b/arch/x86/boot/compressed/unaccepted_memory.c index 17b70627b0cd..818d32169eef 100644 --- a/arch/x86/boot/compressed/unaccepted_memory.c +++ b/arch/x86/boot/compressed/unaccepted_memory.c @@ -13,8 +13,7 @@ void mark_unaccepted(struct boot_params *params, u64 start, u64 num) unsigned int npages; if ((start & PMD_MASK) == (end & PMD_MASK)) { - npages = (end - start) / PAGE_SIZE; - __accept_memory(start, start + npages * PAGE_SIZE); + __accept_memory(start, end); return; } diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index 4d5810c8fab7..1e56d76ca474 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -19,6 +19,11 @@ struct page; #include + +#ifdef CONFIG_UNACCEPTED_MEMORY +#include +#endif + extern struct range pfn_mapped[]; extern int nr_pfn_mapped; diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h index f1f835d3cd78..712128760131 100644 --- a/arch/x86/include/asm/unaccepted_memory.h +++ b/arch/x86/include/asm/unaccepted_memory.h @@ -6,9 +6,12 @@ #include struct boot_params; +struct page; void mark_unaccepted(struct boot_params *params, u64 start, u64 num); void accept_memory(phys_addr_t start, phys_addr_t end); +void maybe_set_page_offline(struct page *page, unsigned int order); +void clear_page_offline(struct page *page, unsigned int order); #endif diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index b31cb52bf1bd..fe4e16322868 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -57,3 +57,5 @@ obj-$(CONFIG_X86_MEM_ENCRYPT_COMMON) += mem_encrypt_common.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o + +obj-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o diff --git a/arch/x86/mm/unaccepted_memory.c b/arch/x86/mm/unaccepted_memory.c new file mode 100644 index 000000000000..e11933f62ead --- /dev/null +++ b/arch/x86/mm/unaccepted_memory.c @@ -0,0 +1,80 @@ +#include +#include +#include +#include + +#include +#include +#include + +static DEFINE_SPINLOCK(unaccepted_memory_lock); + +#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) + +static void __accept_memory(phys_addr_t start, phys_addr_t end) +{ + unsigned long *unaccepted_memory; + unsigned int rs, re; + + unaccepted_memory = __va(boot_params.unaccepted_memory); + bitmap_for_each_set_region(unaccepted_memory, rs, re, + start / PMD_SIZE, + DIV_ROUND_UP(end, PMD_SIZE)) { + /* Platform-specific memory-acceptance call goes here */ + panic("Cannot accept memory"); + bitmap_clear(unaccepted_memory, rs, re - rs); + } +} + +void accept_memory(phys_addr_t start, phys_addr_t end) +{ + if (!boot_params.unaccepted_memory) + return; + + spin_lock(&unaccepted_memory_lock); + __accept_memory(start, end); + spin_unlock(&unaccepted_memory_lock); +} + +void __init maybe_set_page_offline(struct page *page, unsigned int order) +{ + unsigned long *unaccepted_memory; + phys_addr_t addr = page_to_phys(page); + bool unaccepted = true; + unsigned int i; + + if (!boot_params.unaccepted_memory) + return; + + unaccepted_memory = __va(boot_params.unaccepted_memory); + spin_lock(&unaccepted_memory_lock); + if (order < PMD_ORDER) { + BUG_ON(test_bit(addr / PMD_SIZE, unaccepted_memory)); + goto out; + } + + for (i = 0; i < (1 << (order - PMD_ORDER)); i++) { + if (!test_bit(addr / PMD_SIZE + i, unaccepted_memory)) { + unaccepted = false; + break; + } + } + + if (unaccepted) + __SetPageOffline(page); + else + __accept_memory(addr, addr + (PAGE_SIZE << order)); +out: + spin_unlock(&unaccepted_memory_lock); +} + +void clear_page_offline(struct page *page, unsigned int order) +{ + phys_addr_t addr = page_to_phys(page); + + /* PageOffline() page on a free list, but no unaccepted memory? Hm. */ + WARN_ON_ONCE(!boot_params.unaccepted_memory); + + accept_memory(addr, addr + (PAGE_SIZE << order)); + __ClearPageOffline(page); +} From patchwork Tue Aug 10 06:26:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12428063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD1ABC432BE for ; Tue, 10 Aug 2021 06:26:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 600FB61051 for ; Tue, 10 Aug 2021 06:26:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 600FB61051 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 27F888D0002; Tue, 10 Aug 2021 02:26:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F4E98D0003; Tue, 10 Aug 2021 02:26:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E74736B0074; Tue, 10 Aug 2021 02:26:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id C50306B0074 for ; Tue, 10 Aug 2021 02:26:28 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7617C1F054 for ; Tue, 10 Aug 2021 06:26:28 +0000 (UTC) X-FDA: 78458186856.17.C95E135 Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by imf28.hostedemail.com (Postfix) with ESMTP id 328A0900184C for ; Tue, 10 Aug 2021 06:26:28 +0000 (UTC) Received: by mail-lj1-f176.google.com with SMTP id x9so25437149ljj.2 for ; Mon, 09 Aug 2021 23:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BWJUZP0dbskrNbtaJwK/Tbp+8mAAWASI6DAzHFq/lFg=; b=aVjEJN6SQwrYycsPycBz2KNKkeTGZwuzwnHT1R1uplQFzzkx3iDN7NBneR19HXIwuF NXIv8lWfs6zpG/K7Sux3uy9JSAPf3V7yVMRaZZZLrzrEHPvrJapDmNr3vVx2FOgj82De rZe/gFeTDL/bZ2kqUJ067TKudI5KVLZJz9gKXoDS9TYMqIjIqWXgsmguHUYEsfndZeJK JvbzHkO0CUCxjrS+0iXhg6+S5oFl8YWCh1zURsxV+nAHwhlfDyYkwLd/ZkilLCWXkunw 6NATuUgbbguSRL1DrDkg0ne2oRm459dbrn1hVjiOP3qDLvzeCpJq6Y1/2EQqTbh9iGtk ySDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BWJUZP0dbskrNbtaJwK/Tbp+8mAAWASI6DAzHFq/lFg=; b=i2xvpQkqGgyx1otZk4xnAWqHttZJYvS/n7fn0FshLZciwfx/TTshjDC/MsvQZFKZVW uDNB80ixpkPUf5/CP8ErqXintpT3LuMX+bBBfDvvvCQYf+0vEJuW3U7eusAYUu/BU6Yu uI2uXG/fyNkytQ3ZLN6imL4MXdmNtG4h29ev82rLCcqCtUfuRTk9xmbToFW4uNOMVBPM hXshRYPYU6cyIzX6EvuKBGGIikd25c1a1xwTOonVvNbUbexinukH8zJeu+v8lKqEaj2H 9xtMRjjftzxchU5TMUyVPIO6GKgzhayQ4eojlmwSnwQKMxb1tQhf8TbETFI8GY4la2O1 Qs7g== X-Gm-Message-State: AOAM533TJoeIlqT2FGgCXzGP0KnMSGgKMunV7iqrwmNCFYTIIlxu+60m Y1yBOmyXULML80UHNw3VjGEEbQ== X-Google-Smtp-Source: ABdhPJxFXIZNu8BRH3d+/+nzJFBbH2eGlX5f7qfMtPHplWcg+ebB0Wn91QuSk/T4ncwIyHb4y7E2Tw== X-Received: by 2002:a2e:d1a:: with SMTP id 26mr18928460ljn.360.1628576786905; Mon, 09 Aug 2021 23:26:26 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id u3sm1600860lfu.151.2021.08.09.23.26.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Aug 2021 23:26:26 -0700 (PDT) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id B7058102BDC; Tue, 10 Aug 2021 09:26:34 +0300 (+03) To: Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 5/5] x86/tdx: Unaccepted memory support Date: Tue, 10 Aug 2021 09:26:26 +0300 Message-Id: <20210810062626.1012-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> References: <20210810062626.1012-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 328A0900184C Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=shutemov-name.20150623.gappssmtp.com header.s=20150623 header.b=aVjEJN6S; dmarc=none; spf=none (imf28.hostedemail.com: domain of kirill@shutemov.name has no SPF policy when checking 209.85.208.176) smtp.mailfrom=kirill@shutemov.name X-Stat-Signature: asogjyzcaiea5g1b8ouzarik9bdhhgcb X-HE-Tag: 1628576788-998543 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All preparation is complete. Hookup TDX-specific code to accept memory. There are two tdg_accept_memory() implementations: one in main kernel and one in the decompresser. The implementation in core kernel uses tdx_hcall_gpa_intent(). The helper is not available in the decompresser, self-contained implementation added there instead. Note that tdx_hcall_gpa_intent() is going to be more complex once we teach it to accept in 1G and 2M chunks. Signed-off-by: Kirill A. Shutemov --- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/tdx.c | 29 ++++++++++++++++++++ arch/x86/boot/compressed/unaccepted_memory.c | 5 +++- arch/x86/include/asm/tdx.h | 2 ++ arch/x86/kernel/tdx.c | 8 ++++++ arch/x86/mm/unaccepted_memory.c | 6 +++- 6 files changed, 49 insertions(+), 2 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index c0261cc72449..5b1b3dc84c7d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -874,6 +874,7 @@ config INTEL_TDX_GUEST select ARCH_HAS_PROTECTED_GUEST select X86_MEM_ENCRYPT_COMMON select ARCH_HAS_RESTRICTED_VIRTIO_MEMORY_ACCESS + select UNACCEPTED_MEMORY help Provide support for running in a trusted domain on Intel processors equipped with Trusted Domain eXtensions. TDX is a new Intel diff --git a/arch/x86/boot/compressed/tdx.c b/arch/x86/boot/compressed/tdx.c index 88ed6465405b..6f13bdaf327f 100644 --- a/arch/x86/boot/compressed/tdx.c +++ b/arch/x86/boot/compressed/tdx.c @@ -5,6 +5,10 @@ #include "../cpuflags.h" #include "../string.h" +#include "error.h" + +#include +#include #define TDX_CPUID_LEAF_ID 0x21 @@ -32,3 +36,28 @@ bool early_is_tdx_guest(void) return !!tdx_guest; } + +#define TDACCEPTPAGE 6 +#define TDVMCALL_MAP_GPA 0x10001 + +void tdg_accept_memory(phys_addr_t start, phys_addr_t end) +{ + struct tdx_hypercall_output outl = {0}; + int i; + + if (__tdx_hypercall(TDX_HYPERCALL_STANDARD, TDVMCALL_MAP_GPA, + start, end, 0, 0, &outl)) { + error("Cannot accept memory: MapGPA failed\n"); + } + + /* + * For shared->private conversion, accept the page using TDACCEPTPAGE + * TDX module call. + */ + for (i = 0; i < (end - start) / PAGE_SIZE; i++) { + if (__tdx_module_call(TDACCEPTPAGE, start + i * PAGE_SIZE, + 0, 0, 0, NULL)) { + error("Cannot accept memory: page accept failed\n"); + } + } +} diff --git a/arch/x86/boot/compressed/unaccepted_memory.c b/arch/x86/boot/compressed/unaccepted_memory.c index 818d32169eef..146a3a6968a8 100644 --- a/arch/x86/boot/compressed/unaccepted_memory.c +++ b/arch/x86/boot/compressed/unaccepted_memory.c @@ -4,7 +4,10 @@ static inline void __accept_memory(phys_addr_t start, phys_addr_t end) { /* Platform-specific memory-acceptance call goes here */ - error("Cannot accept memory"); + if (early_is_tdx_guest()) + tdg_accept_memory(start, end); + else + error("Cannot accept memory"); } void mark_unaccepted(struct boot_params *params, u64 start, u64 num) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index af6e4cd8078d..f74b6cfde205 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -97,6 +97,8 @@ extern phys_addr_t tdg_shared_mask(void); extern int tdx_hcall_gpa_intent(phys_addr_t gpa, int numpages, enum tdx_map_type map_type); +extern void tdg_accept_memory(phys_addr_t start, phys_addr_t end); + int tdx_mcall_tdreport(u64 data, u64 reportdata); int tdx_mcall_rtmr_extend(u64 data, u64 rmtr); diff --git a/arch/x86/kernel/tdx.c b/arch/x86/kernel/tdx.c index dc6d9441f3be..75f3804f86da 100644 --- a/arch/x86/kernel/tdx.c +++ b/arch/x86/kernel/tdx.c @@ -372,6 +372,14 @@ int tdx_hcall_gpa_intent(phys_addr_t gpa, int numpages, return 0; } +void tdg_accept_memory(phys_addr_t start, phys_addr_t end) +{ + if (tdx_hcall_gpa_intent(start, (end - start) / PAGE_SIZE, + TDX_MAP_PRIVATE)) { + panic("Accepting memory failed\n"); + } +} + static __cpuidle void tdg_halt(void) { u64 ret; diff --git a/arch/x86/mm/unaccepted_memory.c b/arch/x86/mm/unaccepted_memory.c index e11933f62ead..19e0309e128b 100644 --- a/arch/x86/mm/unaccepted_memory.c +++ b/arch/x86/mm/unaccepted_memory.c @@ -5,6 +5,7 @@ #include #include +#include #include static DEFINE_SPINLOCK(unaccepted_memory_lock); @@ -21,7 +22,10 @@ static void __accept_memory(phys_addr_t start, phys_addr_t end) start / PMD_SIZE, DIV_ROUND_UP(end, PMD_SIZE)) { /* Platform-specific memory-acceptance call goes here */ - panic("Cannot accept memory"); + if (prot_guest_has(PATTR_GUEST_TDX)) + tdg_accept_memory(rs * PMD_SIZE, re * PMD_SIZE); + else + panic("Cannot accept memory"); bitmap_clear(unaccepted_memory, rs, re - rs); } }