From patchwork Thu May 30 21:53:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969265 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3A3014C0 for ; Thu, 30 May 2019 21:53:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B1F1428C54 for ; Thu, 30 May 2019 21:53:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A359728C19; Thu, 30 May 2019 21:53:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3009E28A61 for ; Thu, 30 May 2019 21:53:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 380AF6B026A; Thu, 30 May 2019 17:53:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 332776B026B; Thu, 30 May 2019 17:53:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 247276B026D; Thu, 30 May 2019 17:53:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by kanga.kvack.org (Postfix) with ESMTP id ECC076B026A for ; Thu, 30 May 2019 17:53:45 -0400 (EDT) Received: by mail-ot1-f69.google.com with SMTP id q19so2027841otf.0 for ; Thu, 30 May 2019 14:53:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=sJuZdrZ6VMk79b1ZhahuZrudd0+Kp6YlC0eBwgdHqtI=; b=Echon3YegnVwSBa9oXnYYDzLexBx/lWm98c9SgpEg6F4PC/ZnvUPbhbKoW1stO6dTy XkbE8FyP1Z9r2YKI0a0cLFGFRre7AlzJULnTVWNDvDikM4eLl6xFiFIfl2/xHpjxhzL7 dJIwu9DvbuAz/0qSRJ5BI/cE5JTNiRF8hXTT7XnZOdfJ2nVh3N7Dy75+Xea7eKecs/iq DdtbsZvN/BQChCgd7XM/7hxLd1IByRtmBWJ8N7NuWNyJ61zpjIA5tB+EtXHm+KrrcEtC t14OOj5DqkDTj+dDKd0ijhVehIvNe18jD0eyDSg0xisBX6F9FwypJH+iMbqneuvoyiyP 7T4w== X-Gm-Message-State: APjAAAUP8SALxcehfTwOXsHAruvtQ1c0J76gnZA8rxBun5Y2IPHtd8YU QkYew8EABz1PHSPoT2RFimXioaUuquetgqfzvwkMzIeGGp1uUnAS4b2qqSK3Rtn0X+R1jHTrkYV /pzU863k3IbPrtM3BFZ7qP90vWLNHPo6N2MqOIfYQqqIt3neAp6HJCzmkBcOofEk/XA== X-Received: by 2002:a9d:5511:: with SMTP id l17mr4539863oth.158.1559253225614; Thu, 30 May 2019 14:53:45 -0700 (PDT) X-Received: by 2002:a9d:5511:: with SMTP id l17mr4539821oth.158.1559253224568; Thu, 30 May 2019 14:53:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253224; cv=none; d=google.com; s=arc-20160816; b=bGarTUoQI0hfrUrD1Z5o/XTGApoMwrSfMhdE8GzOCh4Vr78AU03SigpQ45oIEasTmr oAb1AFTwPX3aSxBNQNDiidxelKY0q1Xeeo/qVaq17BGw3Zk1/iOjHZ5HPnGTnafxzsnf 6V1PSsx2xLXD5jKMfUW4TmDSCyA0ZEVt732EAikKKj5B5G3oGAwWbVFmdMI752qRFHF+ SlhhVjCiamatlB6Q5jq407hjoi9tWHJ7e3i1ya24oZdicuzjXL9Cb4eOQ+uZHpS4aAhW upkGQJdPBxv76UyY/pOACsRG7DEDjZLPjY2mYVZoBvVwB7WeyzZpOXsWwUMcoPFaSBi7 9/4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=sJuZdrZ6VMk79b1ZhahuZrudd0+Kp6YlC0eBwgdHqtI=; b=VPhxxEHMg/SBVdB7zn6RteKSQ9NpDnhJjO84mOyBFdA4s1R87ynhIB5gfkOtLzjDhB IaODIrHRupiDiD8E4OYloX+nz3QeU82YQITvSlZbrA/a0oTtsnScwCSjwe8yhe5pFOEC cNGY3S/WR7GmQYf5YUoJNqXYw8uiEvytBbSKN7E0jrDp3a4JZNjvTASR+ISU4VDA0+Rp PqfhJYklFE0WPvST6CJKexbRgnufgNkmhrt39c4ISF+UiyvuuTiDjGQezNjURVx8n+3B mMaTLg2TIqrV2dm/khAVJyA84jswVZy6J/uFjMdHJRoaGiZ08e3UbCc32cMuRJc1BFa1 vAGQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=PsxZY9cX; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c10sor1961667otb.23.2019.05.30.14.53.44 for (Google Transport Security); Thu, 30 May 2019 14:53:44 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=PsxZY9cX; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=sJuZdrZ6VMk79b1ZhahuZrudd0+Kp6YlC0eBwgdHqtI=; b=PsxZY9cXHZU3UHhQbhmWYsY+UJaHm8siNxxjBiful25kOQ+QKikMo6MgAVTCdgUVRf HjBxf/inGy+2U2z+5S+aaxGx2eUrk6bqqatAILgaaLwR1X4LkQlo7YDmx157FhcBU5Dh SKPTMz5awBl/iOy0G0otyV+PrTYLeu7JgLh7vznhqdFmuP7VyNkvH+QRBD6mjV8dj0ac VsIz6MtfFh0yXHkuy6sjpNia/mdmJuFWmSloJd2uK3Gf8a3fao/2BY0tS567OpBezrz+ nyjzchNz30eSf80CDrDHUso4ILvcPw8/p2r5zT3iwCzx5MLu0IwOQEdUzi8g7aVFn/P6 dHzg== X-Google-Smtp-Source: APXvYqxMsojOf0egOxfGWLBhthd5R18VJ7E3zUiMlsMMhMaEmJH88aU+eNQTsegxUV9DbttxV/LJgw== X-Received: by 2002:a9d:1b6d:: with SMTP id l100mr4256814otl.15.1559253224151; Thu, 30 May 2019 14:53:44 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id v89sm1441749otb.14.2019.05.30.14.53.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:53:43 -0700 (PDT) Subject: [RFC PATCH 01/11] mm: Move MAX_ORDER definition closer to pageblock_order From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:53:41 -0700 Message-ID: <20190530215341.13974.19456.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck The definition of MAX_ORDER being contained in mmzone.h is problematic when wanting to just get access to things like pageblock_order since pageblock_order is defined on some architectures as being based on MAX_ORDER and it isn't included in pageblock-flags.h. Move the definition of MAX_ORDER into pageblock-flags.h so that it is defined in the same header as pageblock_order. By doing this we don't need to also include mmzone.h. The definition of MAX_ORDER will still be accessible to any file that includes mmzone.h as it includes pageblock-flags.h. Signed-off-by: Alexander Duyck --- include/linux/mmzone.h | 8 -------- include/linux/pageblock-flags.h | 8 ++++++++ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 70394cabaf4e..a6bdff538437 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -22,14 +22,6 @@ #include #include -/* Free memory management - zoned buddy allocator. */ -#ifndef CONFIG_FORCE_MAX_ZONEORDER -#define MAX_ORDER 11 -#else -#define MAX_ORDER CONFIG_FORCE_MAX_ZONEORDER -#endif -#define MAX_ORDER_NR_PAGES (1 << (MAX_ORDER - 1)) - /* * PAGE_ALLOC_COSTLY_ORDER is the order at which allocations are deemed * costly to service. That is between allocation orders which should diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h index 06a66327333d..e9e8006ccae1 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -40,6 +40,14 @@ enum pageblock_bits { NR_PAGEBLOCK_BITS }; +/* Free memory management - zoned buddy allocator. */ +#ifndef CONFIG_FORCE_MAX_ZONEORDER +#define MAX_ORDER 11 +#else +#define MAX_ORDER CONFIG_FORCE_MAX_ZONEORDER +#endif +#define MAX_ORDER_NR_PAGES (1 << (MAX_ORDER - 1)) + #ifdef CONFIG_HUGETLB_PAGE #ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE From patchwork Thu May 30 21:53:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969269 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53C4A912 for ; Thu, 30 May 2019 21:53:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43D5C28958 for ; Thu, 30 May 2019 21:53:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 422E328C46; Thu, 30 May 2019 21:53:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7797728958 for ; Thu, 30 May 2019 21:53:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 814556B026D; Thu, 30 May 2019 17:53:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7C5D76B026E; Thu, 30 May 2019 17:53:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B37A6B026F; Thu, 30 May 2019 17:53:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by kanga.kvack.org (Postfix) with ESMTP id 41B4F6B026D for ; Thu, 30 May 2019 17:53:53 -0400 (EDT) Received: by mail-oi1-f198.google.com with SMTP id k66so2743390oib.20 for ; Thu, 30 May 2019 14:53:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=7/ShmDor402LGxJdUnB6r3jYiu3C2H9J7GnwmwTdNWw=; b=kg2WCxwO/fVnAnRqa7T55kH7hBfSGPuvDGLxcIfZp6hM1Fd50lOStELZjuFXguOwEF GkvikoGUx9IPiLk+P9BSVX17+aGhpZkV2Xlf5l3wfZ9A5J01Rd3H2V2zg8BdvaDRrHo0 iMbxUqtFqg41KlYV3LdKQOxyFtNhiLVHKF5rUzA/R428/rs8nsw6QsvYag0EdElE2eha 8W5/aUn9cJvXoSmSb1YVXB+qsjSC0KDcr0iLCocAX9x0pfmlskdFzqPfJdhifkLnwTi+ VGfshYoOhuqI0kMSmKL+e4zg8pnJ7UpsJ7ebNL1rj847INipthftmNJYV2TgS2tTuuS7 7RrA== X-Gm-Message-State: APjAAAWBhdtipW4fjLBb92yqoYFIzQIMw2bXfqiAcYNf3pK09C5HGFkd EeDfpamnZSgaOzkopic7LIoWGOCCUOTAVNlfclgin9cyKG7bAHXCDGqJT2em2isic38VaM2T+j3 16xZ3liOF+VBvi5Xv9etwd9fFGQN/0+zKCaoHrGtd+oIupxRl53xJ0szQmPzM9PWjYw== X-Received: by 2002:a05:6830:2054:: with SMTP id f20mr4429355otp.288.1559253232818; Thu, 30 May 2019 14:53:52 -0700 (PDT) X-Received: by 2002:a05:6830:2054:: with SMTP id f20mr4429320otp.288.1559253231953; Thu, 30 May 2019 14:53:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253231; cv=none; d=google.com; s=arc-20160816; b=vizsME4AAUReKSlRpZVq0kEcQiPG3sfDOXYaoGX3AyvDCWBdYUxYh/WHETPPMYij16 rHHLPh4aPVyCoDumnyg9Q/SCRZQMJVasTrwIfDwFxxfPQZWvZOFC+ZNwUSvsm3D4+VL2 veHu61T8dKGkenJAn+ZwQk55aTuRFRrKN4MU8xHGMJjohgYBUocKO+KHgJXsUHMwTh5d uEXAca1RUZz2iL0SD16NoOUw4iKC5OTChcTZ9KlugWCbTdElYPOOYlJ4HEza+cIhFAl8 8QxOK6+r+S6O6kNiBjdwRwUQ9OgevRUyujJ1lLrQ1d20i4+/ADdzgnReCDnD5h2uSp6W 4CxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=7/ShmDor402LGxJdUnB6r3jYiu3C2H9J7GnwmwTdNWw=; b=CFUNmIhQDt30BmcFVcQAiygxNBSoIdLT07G6QuJNKK+YFtxrokhg5sIHAxKtGd6C9f 4Ep5FsAyWoErsrFtK+IxzcJJoZ76iSvWit3k0HTsodsdxqGHwPecTQIEOG3oZ1Yp4mwV W6WOV0Fv2LotyM1wMa8LnMkEVAqtFvy8PsxuZlzKbuPjMbHjYb4cpzI92PE0crz7axWt IW2sTgF39gNEUprtYusjgkxKy4n74TQfFskrvBfx22pyWHwcJL+ZG3/buCQLVN/+NHm4 gJ49xIWjeR+U41FvZ2R+5JI1KRDckcloI1jCULVd7OeM+PMqu6jFCvEjKiUn+avezg2p fvTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="jad/iZgD"; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p5sor989869oia.156.2019.05.30.14.53.51 for (Google Transport Security); Thu, 30 May 2019 14:53:51 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="jad/iZgD"; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=7/ShmDor402LGxJdUnB6r3jYiu3C2H9J7GnwmwTdNWw=; b=jad/iZgDNFkLMXFeonjDMysadIOYcRNWeBjMK29xpgVosay9LzT2MFQMnv8Astm+6I pgM0i8wpDmzoi9qt7K91vUNWAGuh+XGZ2oJLqx8pewNeoPkJHZ5l/7H/8Ysxd6AtsFFl aCj2g9XNTxsTLctkn9rrNH39jDVy6ARaVZzwtxvqbbMedEezNh4AZEXOWhWMzzldDSdV apibBvcrrTo43whDeU+YJ3/f0h4B9swIqxrcA+2OL8YXB2+ysoauJw7hhuxHQ0jgxIeu X2GG8rL6oO7EhHvgrSOsaIBDg+IvsM2JE3gktx0c3oymPA5CCfmTN5G5oMzc74r4ZgSx RU+w== X-Google-Smtp-Source: APXvYqxpsklzRzklzaaWunr5R2owZZS1CzYFOUcljqCLnm6wJpxHd7Xotg82bP7Wx08VkZW6UTLEHw== X-Received: by 2002:aca:3242:: with SMTP id y63mr3882561oiy.148.1559253231505; Thu, 30 May 2019 14:53:51 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id t21sm1459319otj.46.2019.05.30.14.53.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:53:50 -0700 (PDT) Subject: [RFC PATCH 02/11] mm: Adjust shuffle code to allow for future coalescing From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:53:49 -0700 Message-ID: <20190530215349.13974.25544.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck This patch is meant to move the head/tail adding logic out of the shuffle code and into the __free_one_page function since ultimately that is where it is really needed anyway. By doing this we should be able to reduce the overhead and can consolidate all of the list addition bits in one spot. Signed-off-by: Alexander Duyck --- include/linux/mmzone.h | 12 -------- mm/page_alloc.c | 70 +++++++++++++++++++++++++++--------------------- mm/shuffle.c | 24 ---------------- mm/shuffle.h | 35 ++++++++++++++++++++++++ 4 files changed, 74 insertions(+), 67 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a6bdff538437..297edb45071a 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -108,18 +108,6 @@ static inline void add_to_free_area_tail(struct page *page, struct free_area *ar area->nr_free++; } -#ifdef CONFIG_SHUFFLE_PAGE_ALLOCATOR -/* Used to preserve page allocation order entropy */ -void add_to_free_area_random(struct page *page, struct free_area *area, - int migratetype); -#else -static inline void add_to_free_area_random(struct page *page, - struct free_area *area, int migratetype) -{ - add_to_free_area(page, area, migratetype); -} -#endif - /* Used for pages which are on another list */ static inline void move_to_free_area(struct page *page, struct free_area *area, int migratetype) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c061f66c2d0c..2fa5bbb372bb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -851,6 +851,36 @@ static inline struct capture_control *task_capc(struct zone *zone) #endif /* CONFIG_COMPACTION */ /* + * If this is not the largest possible page, check if the buddy + * of the next-highest order is free. If it is, it's possible + * that pages are being freed that will coalesce soon. In case, + * that is happening, add the free page to the tail of the list + * so it's less likely to be used soon and more likely to be merged + * as a higher order page + */ +static inline bool +buddy_merge_likely(unsigned long pfn, unsigned long buddy_pfn, + struct page *page, unsigned int order) +{ + struct page *higher_page, *higher_buddy; + unsigned long combined_pfn; + + if (is_shuffle_order(order) || order >= (MAX_ORDER - 2)) + return false; + + if (!pfn_valid_within(buddy_pfn)) + return false; + + combined_pfn = buddy_pfn & pfn; + higher_page = page + (combined_pfn - pfn); + buddy_pfn = __find_buddy_pfn(combined_pfn, order + 1); + higher_buddy = higher_page + (buddy_pfn - combined_pfn); + + return pfn_valid_within(buddy_pfn) && + page_is_buddy(higher_page, higher_buddy, order + 1); +} + +/* * Freeing function for a buddy system allocator. * * The concept of a buddy system is to maintain direct-mapped table @@ -879,11 +909,12 @@ static inline void __free_one_page(struct page *page, struct zone *zone, unsigned int order, int migratetype) { - unsigned long combined_pfn; + struct capture_control *capc = task_capc(zone); unsigned long uninitialized_var(buddy_pfn); - struct page *buddy; + unsigned long combined_pfn; + struct free_area *area; unsigned int max_order; - struct capture_control *capc = task_capc(zone); + struct page *buddy; max_order = min_t(unsigned int, MAX_ORDER, pageblock_order + 1); @@ -952,35 +983,12 @@ static inline void __free_one_page(struct page *page, done_merging: set_page_order(page, order); - /* - * If this is not the largest possible page, check if the buddy - * of the next-highest order is free. If it is, it's possible - * that pages are being freed that will coalesce soon. In case, - * that is happening, add the free page to the tail of the list - * so it's less likely to be used soon and more likely to be merged - * as a higher order page - */ - if ((order < MAX_ORDER-2) && pfn_valid_within(buddy_pfn) - && !is_shuffle_order(order)) { - struct page *higher_page, *higher_buddy; - combined_pfn = buddy_pfn & pfn; - higher_page = page + (combined_pfn - pfn); - buddy_pfn = __find_buddy_pfn(combined_pfn, order + 1); - higher_buddy = higher_page + (buddy_pfn - combined_pfn); - if (pfn_valid_within(buddy_pfn) && - page_is_buddy(higher_page, higher_buddy, order + 1)) { - add_to_free_area_tail(page, &zone->free_area[order], - migratetype); - return; - } - } - - if (is_shuffle_order(order)) - add_to_free_area_random(page, &zone->free_area[order], - migratetype); + area = &zone->free_area[order]; + if (buddy_merge_likely(pfn, buddy_pfn, page, order) || + is_shuffle_tail_page(order)) + add_to_free_area_tail(page, area, migratetype); else - add_to_free_area(page, &zone->free_area[order], migratetype); - + add_to_free_area(page, area, migratetype); } /* diff --git a/mm/shuffle.c b/mm/shuffle.c index 3ce12481b1dc..55d592e62526 100644 --- a/mm/shuffle.c +++ b/mm/shuffle.c @@ -4,7 +4,6 @@ #include #include #include -#include #include #include "internal.h" #include "shuffle.h" @@ -182,26 +181,3 @@ void __meminit __shuffle_free_memory(pg_data_t *pgdat) for (z = pgdat->node_zones; z < pgdat->node_zones + MAX_NR_ZONES; z++) shuffle_zone(z); } - -void add_to_free_area_random(struct page *page, struct free_area *area, - int migratetype) -{ - static u64 rand; - static u8 rand_bits; - - /* - * The lack of locking is deliberate. If 2 threads race to - * update the rand state it just adds to the entropy. - */ - if (rand_bits == 0) { - rand_bits = 64; - rand = get_random_u64(); - } - - if (rand & 1) - add_to_free_area(page, area, migratetype); - else - add_to_free_area_tail(page, area, migratetype); - rand_bits--; - rand >>= 1; -} diff --git a/mm/shuffle.h b/mm/shuffle.h index 777a257a0d2f..3f4edb60a453 100644 --- a/mm/shuffle.h +++ b/mm/shuffle.h @@ -3,6 +3,7 @@ #ifndef _MM_SHUFFLE_H #define _MM_SHUFFLE_H #include +#include /* * SHUFFLE_ENABLE is called from the command line enabling path, or by @@ -43,6 +44,35 @@ static inline bool is_shuffle_order(int order) return false; return order >= SHUFFLE_ORDER; } + +static inline bool is_shuffle_tail_page(int order) +{ + static u64 rand; + static u8 rand_bits; + u64 rand_old; + + if (!is_shuffle_order(order)) + return false; + + /* + * The lack of locking is deliberate. If 2 threads race to + * update the rand state it just adds to the entropy. + */ + if (rand_bits-- == 0) { + rand_bits = 64; + rand = get_random_u64(); + } + + /* + * Test highest order bit while shifting our random value. This + * should result in us testing for the carry flag following the + * shift. + */ + rand_old = rand; + rand <<= 1; + + return rand < rand_old; +} #else static inline void shuffle_free_memory(pg_data_t *pgdat) { @@ -60,5 +90,10 @@ static inline bool is_shuffle_order(int order) { return false; } + +static inline bool is_shuffle_tail_page(int order) +{ + return false; +} #endif #endif /* _MM_SHUFFLE_H */ From patchwork Thu May 30 21:53:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969273 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A71861575 for ; Thu, 30 May 2019 21:54:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99FA128C19 for ; Thu, 30 May 2019 21:54:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8E20928A5B; Thu, 30 May 2019 21:54:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 227BF28B01 for ; Thu, 30 May 2019 21:54:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A85746B026F; Thu, 30 May 2019 17:54:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A35C46B0270; Thu, 30 May 2019 17:54:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94C896B0271; Thu, 30 May 2019 17:54:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by kanga.kvack.org (Postfix) with ESMTP id 6A42C6B026F for ; Thu, 30 May 2019 17:54:00 -0400 (EDT) Received: by mail-oi1-f197.google.com with SMTP id v13so2755393oie.12 for ; Thu, 30 May 2019 14:54:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=Wtqhuc8tld+njwpNjaUC3LbpBQWLP9ZV0hs8fIq5ASw=; b=Rd0L7nFpwelfCnapnBea/xND+oTFmL4khjQhwid/F4pw8F3O/ha1e9HiFUOO4ewEcW Z4Hal1JQkZjIaD2tmcFX6yyzIgzUCZ2jHVMHGP163xFHma2sgX95NFpgqqURUST9mcM+ pfKxvXV9JYX3ZG1rQ+qkVU2jH/fN0ayS/68X8uy7+6SatZXfMZDgyM4t84aIsfVvsasx LKCW6AIbu30pypr6sk/emze5UpT2HiHBCneFtm0lptkRftmXauTLspcgmQL2NqPlVZn5 RGalKuG9GXO1fNscseTmLi9eoS8RTqLGazWOcK8VxeB5UInAsDjMUa0ZPjVJpyTmM6I3 ctIA== X-Gm-Message-State: APjAAAUDl2opuN56NCxuGi2WT3WKIcJpOKkRBhSNkpQqLP/Dg0QLMY8A JdMVKOhOUH1+u20IEPXVhsH2OAqGfcGt3fPGZb15dhZVtwIaiRXwf4LyJ7r0Bguj+FDyPhnn3wW HFh3lQyRFjdSFI7Z607Lrrfdh2/kDwYMtG8B9SgkW3pNxmn2SjexyaG3tbIMBjm9gLw== X-Received: by 2002:aca:300d:: with SMTP id w13mr4006782oiw.26.1559253240117; Thu, 30 May 2019 14:54:00 -0700 (PDT) X-Received: by 2002:aca:300d:: with SMTP id w13mr4006765oiw.26.1559253239414; Thu, 30 May 2019 14:53:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253239; cv=none; d=google.com; s=arc-20160816; b=M90+IlqJnKszUyjJQcPJtkXwIc2SCjUOCLMoiMGMuc5oURcKyi9pkHkgdzNstWjEdD rOUZmSqKc4kbX19rmZuHtELDEDk8HS7XnI/M3rGqmddXgTnoZwUfP8Sd7Ie+q3vBxSXr 94GOA26qEYhEnqHVqyglQk7UmpWDTxNmGfCqxsrEgulU/ZJmUz1cOBZF9uxtA6LPL8Mm ZdTYmig0X7chUOQW8Xf/sXlTNM5w7PabVOvuUq1RV0OQLk2XKd4Kc5M1m/zYK1DSEnPi Mwkt2Ekp60UE4V0ZIQqpFssHKV5OAxFluPqAmWrqZUjg9YlLn78fdpGapktxnBKFHjDf BWVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=Wtqhuc8tld+njwpNjaUC3LbpBQWLP9ZV0hs8fIq5ASw=; b=GpkDJCeWsCJlWCLza1XtqIEEgEczXp1AS68MZX6Cr8EIKT22YkH5YM/l6YewBgtuca Kul5dhe1WPb9PoxjILkJGjsq2SuHbR7J/X2QhUO9vxyNdAGvKU0+6uFA1rBbTwvl+IhU GArMOjVRhuVOPnz8n5Kxy1ei1m7ccbCp7oXXUt/ArtN2xO47OADsK6gSMSq6taHqWYmu iwM1+0yIhNJOwRzK9S10/XQCSlFs2hgQsybuugQandf2eZ+umtshwOHDp8bMDve497DY b13AUUN9F/zkc4kCd5oBWttTXeGXMBJxSHecf7jy0tdSUt8kqVXn+ue+yHXWg/UdWLI9 M8JQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=XJvjTBc1; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c26sor1978166otr.58.2019.05.30.14.53.59 for (Google Transport Security); Thu, 30 May 2019 14:53:59 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=XJvjTBc1; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=Wtqhuc8tld+njwpNjaUC3LbpBQWLP9ZV0hs8fIq5ASw=; b=XJvjTBc1+3PwDTickpM7WozJzbnV4CpwNeBr8/C7ZbJbctSTJZ+22GmcTknusuOoDr 2D/QL0yQASYrqIjKmMGZ3xqFG90FA2JAtq+lrDy0HQr9MXSz2qBD2cBdM4ZhHAuwE/Zd Q9dwWl8Ada9Zdo9/JEx9uFscSKEAWOpBTfgK529lzzWDXo6fUzrmPi2d62PgdJ29IGOa gK7cUg9CxpeecwgzN6TbiRfynxidJ0jNKZW5/jR6vInd5DleH5Nq3BWGpnZFvrhdNCBK zATeaXk0EmpgYeqnWbNJyZPrTHkkI1urnEhXtBXMhTPcPWMDJBMr1rxPelkrhnsQoH8+ rpuQ== X-Google-Smtp-Source: APXvYqzVcIJEh7xNp7JK47qwuZzzsGLJK5TMq9FSbM20Pm3+szfoUg8qiY/3svza+IjLBI7oHuRCkg== X-Received: by 2002:a9d:7347:: with SMTP id l7mr4421138otk.183.1559253239042; Thu, 30 May 2019 14:53:59 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id o124sm1462102oig.23.2019.05.30.14.53.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:53:58 -0700 (PDT) Subject: [RFC PATCH 03/11] mm: Add support for Treated Buddy pages From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:53:56 -0700 Message-ID: <20190530215356.13974.95767.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck This patch is adding support for flagging pages as "Treated" within the buddy allocator. If memory aeration is not enabled then the value will always be treated as false and the set/clear operations will have no effect. Signed-off-by: Alexander Duyck --- include/linux/mmzone.h | 1 + include/linux/page-flags.h | 32 ++++++++++++++++++++++++++++++++ mm/page_alloc.c | 5 +++++ 3 files changed, 38 insertions(+) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 297edb45071a..0263d5bf0b84 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -127,6 +127,7 @@ static inline void del_page_from_free_area(struct page *page, { list_del(&page->lru); __ClearPageBuddy(page); + __ResetPageTreated(page); set_page_private(page, 0); area->nr_free--; } diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 9f8712a4b1a5..1f8ccb98dd69 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -722,12 +722,32 @@ static inline int page_has_type(struct page *page) VM_BUG_ON_PAGE(!PageType(page, 0), page); \ page->page_type &= ~PG_##lname; \ } \ +static __always_inline void __ResetPage##uname(struct page *page) \ +{ \ + VM_BUG_ON_PAGE(!PageType(page, 0), page); \ + page->page_type |= PG_##lname; \ +} \ static __always_inline void __ClearPage##uname(struct page *page) \ { \ VM_BUG_ON_PAGE(!Page##uname(page), page); \ page->page_type |= PG_##lname; \ } +#define PAGE_TYPE_OPS_DISABLED(uname) \ +static __always_inline int Page##uname(struct page *page) \ +{ \ + return false; \ +} \ +static __always_inline void __SetPage##uname(struct page *page) \ +{ \ +} \ +static __always_inline void __ResetPage##uname(struct page *page) \ +{ \ +} \ +static __always_inline void __ClearPage##uname(struct page *page) \ +{ \ +} + /* * PageBuddy() indicates that the page is free and in the buddy system * (see mm/page_alloc.c). @@ -744,6 +764,18 @@ static inline int page_has_type(struct page *page) PAGE_TYPE_OPS(Offline, offline) /* + * PageTreated() is an alias for Offline, however it is not meant to be an + * exclusive value. It should be combined with PageBuddy() when seen as it + * is meant to indicate that the page has been scrubbed while waiting in + * the buddy system. + */ +#ifdef CONFIG_AERATION +PAGE_TYPE_OPS(Treated, offline) +#else +PAGE_TYPE_OPS_DISABLED(Treated) +#endif + +/* * If kmemcg is enabled, the buddy allocator will set PageKmemcg() on * pages allocated with __GFP_ACCOUNT. It gets cleared on page free. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2fa5bbb372bb..2894990862bd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -942,6 +942,11 @@ static inline void __free_one_page(struct page *page, goto done_merging; if (!page_is_buddy(page, buddy, order)) goto done_merging; + + /* If buddy is not treated, then do not mark page treated */ + if (!PageTreated(buddy)) + __ResetPageTreated(page); + /* * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page, * merge with it and move up one order. From patchwork Thu May 30 21:54:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969279 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA351912 for ; Thu, 30 May 2019 21:54:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8D2328C52 for ; Thu, 30 May 2019 21:54:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AC49D28C45; Thu, 30 May 2019 21:54:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E67A628AB3 for ; Thu, 30 May 2019 21:54:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88A396B0271; Thu, 30 May 2019 17:54:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8399A6B0272; Thu, 30 May 2019 17:54:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 728596B0273; Thu, 30 May 2019 17:54:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f71.google.com (mail-ot1-f71.google.com [209.85.210.71]) by kanga.kvack.org (Postfix) with ESMTP id 48C706B0271 for ; Thu, 30 May 2019 17:54:15 -0400 (EDT) Received: by mail-ot1-f71.google.com with SMTP id z11so3483588otk.7 for ; Thu, 30 May 2019 14:54:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=B4jiG1jHnMkQIVQ7CnmS/QGR53dJGB1zbfKCiInlabY=; b=XAwggdPuGXjofBxdJfTKxlrPfQAV9alXhbtPGjh3Ds32X/8P15PwF7DanUpk4fjm0X bSG5/bD4mY2xyYcu7wn/dMub3RLpelEbjgj81sBwjCvX/GkFz44ZNP69hrAvGh6xvjNA fEPEM1Ob9XV27HX/eflSIZwDUilOP3F7FxaZCjotYBcp3E+Xa1nv/TSJimm0cfBgWG2F Nk+fXy8zWn6vCPGm/U1Ctm7IscPmHvUgLM1mLSqCM0ricCEYAa9p5V8eOG9+m6ym/DYC 4kkEWAY2EcF7Tq8vzeFgOkNPt952Y45LyI3wF/Y5hFEfkZ/N+Pvuqv6eQWL/g0UrK9JJ ds9w== X-Gm-Message-State: APjAAAUcI2egfuynwPrm7coNCnJJZQUo4gUTmOURxi1dYw5THCGSYexA C44pORWgd/DLh9v3dyt2aJ5Oi2eF565q6B/7JYctV2vtc6l95XnWGn9AKOgxQqV/xRqBseGaEVh wGOmnp3ZzpTVJopqsYyZQl8z9nWITZJ5DnquKyJjD2eBbYSbkuwyESaYO3B97C8Zfxg== X-Received: by 2002:aca:e68f:: with SMTP id d137mr2489041oih.14.1559253254919; Thu, 30 May 2019 14:54:14 -0700 (PDT) X-Received: by 2002:aca:e68f:: with SMTP id d137mr2489012oih.14.1559253254165; Thu, 30 May 2019 14:54:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253254; cv=none; d=google.com; s=arc-20160816; b=G4pi/vtXLzqcIUVPLHqCtWb0Ezi1XKJ3mL21VhCsTtd6FaUVmIQMB4/qIYQkNl4Mhj 2DnG0jlSjgmjX3MrwDKov0ABxxcVcWnN6KmeQ2krS41M5pOC2qbdMX8W+worSdnOT8uN dljOIFfiyDGQWdkWWasgBD1eAU2/bnavv53fCnkO6KYmqxXvmFhZru3e6ySkgQCLbYzk sdieXzD6VbfsT9FakyrLADaOFE8PsUg5AJqzQ0cvtpThrM9iFUuxtcAgqoMH5ejqCKLo OABMye5/0IVsKtyFE0j87dseLUFlq5UQHIyPmBemy47WHAi3ehxBgoyrR/Af1vVyJKrf hjtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=B4jiG1jHnMkQIVQ7CnmS/QGR53dJGB1zbfKCiInlabY=; b=vpU23ZzpQOg6NXP8iFRWhMfs7+tk6hnTJt+8owGI59OM6CTTB0GegxjKlWW1e+jkV/ 8me9oZ2a1YFGpXT5OoWE012FNbgtnAP4zz+dhaUx3e1jZKOIUGpKyARA3H7LNZR9SiYc GvZ+fRVP3v2VwcAkPHfcLF/iOvAXmr+3qwI6edS4jMvbRdbdpYSgM+hYrXU/qqrFX+JX YwbSU5cpVt3HeTmudo3rRNeW+qpkvIkMH2h8gvcgQ8S7oJTRPNzQcLoEeo9/Hrp1T6+w J8wP91Cw19e1V78gtgA/XYQJoIhf7BMKLIq5Y9J7tgYZHs60rQTuWoxzjqH/XHc6dVPL gmHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gFqv4vxW; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id r67sor1689468oib.117.2019.05.30.14.54.14 for (Google Transport Security); Thu, 30 May 2019 14:54:14 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gFqv4vxW; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=B4jiG1jHnMkQIVQ7CnmS/QGR53dJGB1zbfKCiInlabY=; b=gFqv4vxWqkG3q53XLUYxfM48NGGzs7jrJLI7d0NMsnFs3NTQJkvh/fhH7mNOhxNcwm Sibve0l8XDYlIMV6OuYkQ6hf2niU43j1ZVnWuTaSLmSd0utr5H6sIS+JtZc+zOty4IZi jjpXbccBXneHJ+bauOZ5a1GEd+4AOlJzJF95ua0OmIFggONVmImQ37q6Gp7FU/ZiUdju aosW9LUpVAmmvtENAf6doZ+0L5J0NvgSXY9lSqm4aatO01j1LJRrnC/71+cwmr56xVyH oy5F4HCSkc722yy8f4DT7yX+LiTTfV7tGD7wAWHkP/5MMiq7ldDQ90Ta2pzoEocPFtCj 4Ffg== X-Google-Smtp-Source: APXvYqwuDRLHPeeQeLx6gytguQrqXX7UFDROgnc7qqhaG7g508ei93QI0YTUhPJGvN6SN4xgMuKEFA== X-Received: by 2002:aca:ec0f:: with SMTP id k15mr3783572oih.43.1559253253781; Thu, 30 May 2019 14:54:13 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id w130sm1429402oib.44.2019.05.30.14.54.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:54:13 -0700 (PDT) Subject: [RFC PATCH 05/11] mm: Propogate Treated bit when splitting From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:54:11 -0700 Message-ID: <20190530215411.13974.73205.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck When we are going to call "expand" to split a page into subpages we should mark those subpages as being "Treated" if the parent page was a "Treated" page. By doing this we can avoid potentially providing hints on a page that was already hinted at a larger page size as being unused. Signed-off-by: Alexander Duyck --- include/linux/mmzone.h | 8 ++++++-- mm/page_alloc.c | 18 +++++++++++++++--- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 988c3094b686..a55fe6d2f63c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -97,16 +97,20 @@ struct free_area { static inline void add_to_free_area(struct page *page, struct free_area *area, int migratetype) { + if (PageTreated(page)) + area->nr_free_treated++; + else + area->nr_free_raw++; + list_add(&page->lru, &area->free_list[migratetype]); - area->nr_free_raw++; } /* Used for pages not on another list */ static inline void add_to_free_area_tail(struct page *page, struct free_area *area, int migratetype) { - list_add_tail(&page->lru, &area->free_list[migratetype]); area->nr_free_raw++; + list_add_tail(&page->lru, &area->free_list[migratetype]); } /* Used for pages which are on another list */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 10eaea762627..f6c067c6c784 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1965,7 +1965,7 @@ void __init init_cma_reserved_pageblock(struct page *page) */ static inline void expand(struct zone *zone, struct page *page, int low, int high, struct free_area *area, - int migratetype) + int migratetype, bool treated) { unsigned long size = 1 << high; @@ -1984,8 +1984,17 @@ static inline void expand(struct zone *zone, struct page *page, if (set_page_guard(zone, &page[size], high, migratetype)) continue; - add_to_free_area(&page[size], area, migratetype); set_page_order(&page[size], high); + if (treated) + __SetPageTreated(&page[size]); + + /* + * The list we are placing this page in should be empty + * so it should be safe to place it here without worrying + * about creating a block of raw pages floating in between + * two blocks of treated pages. + */ + add_to_free_area(&page[size], area, migratetype); } } @@ -2122,6 +2131,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, unsigned int current_order; struct free_area *area; struct page *page; + bool treated; /* Find a page of the appropriate size in the preferred list */ for (current_order = order; current_order < MAX_ORDER; ++current_order) { @@ -2129,8 +2139,10 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, page = get_page_from_free_area(area, migratetype); if (!page) continue; + treated = PageTreated(page); del_page_from_free_area(page, area); - expand(zone, page, order, current_order, area, migratetype); + expand(zone, page, order, current_order, area, migratetype, + treated); set_pcppage_migratetype(page, migratetype); return page; } From patchwork Thu May 30 21:54:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 697B314C0 for ; Thu, 30 May 2019 21:54:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5962928A6A for ; Thu, 30 May 2019 21:54:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 575C528C50; Thu, 30 May 2019 21:54:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B27D728B01 for ; Thu, 30 May 2019 21:54:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09A8A6B0273; Thu, 30 May 2019 17:54:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 04A4B6B0274; Thu, 30 May 2019 17:54:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E07066B0275; Thu, 30 May 2019 17:54:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by kanga.kvack.org (Postfix) with ESMTP id B3C4A6B0273 for ; Thu, 30 May 2019 17:54:22 -0400 (EDT) Received: by mail-ot1-f70.google.com with SMTP id n90so93995otn.22 for ; Thu, 30 May 2019 14:54:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=s26j7V3iEZr//55S8rYqaobNc+2iO706C6wxuCttLf0=; b=lZXyHBkZMNw3FYeWzKStsifnMbyEOOUYdIs6fVDQxVZNEWRUF9hzD+JxP9k3xBIKmt 3DBbCanKYk2fyU0xSXnJycaeWZIIt02OKjgIA6uFylSyx+Wvk3J7smJyEQZ/6E31lIuy Gzu1lkLGOeaRr4o141R1vWTsjTkMcq0/AYd4OQ6ONJKNUd/F/E7aI95ZHX+0OBi7gZRL zKUcgQnjPGuU1JYWvFaNRmcO9xOnD8YXu6RgChF8PGc936YNo+kVZdIGzkLNGC7GLKxm 49mJfBbzGEsB/xTxgg6Dc9Dq/DXWeGacsaBJ0H1pWdJIvQUGzY+XFH1N8DdQ4DIe/MAj NRZg== X-Gm-Message-State: APjAAAUpd9Ayn23zUzm7Sjd1QegMMA+IdQuPGtLrtytwnumQRNqsxCCa Qb7GS+dGbuVTm7QsBC7f2bDtdwq4j6ZFVmTj1CQiQ0U4t53q5lXLZnXI1hoAzb4yy1ngAKo6D1I UtGsMKxAEm7tTqFe2tv+DkqusKlkGlFJ5MIBH4GFYjSu2JiAbSi19nnSkuruJZnPL8g== X-Received: by 2002:aca:ec53:: with SMTP id k80mr53477oih.123.1559253262403; Thu, 30 May 2019 14:54:22 -0700 (PDT) X-Received: by 2002:aca:ec53:: with SMTP id k80mr53446oih.123.1559253261540; Thu, 30 May 2019 14:54:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253261; cv=none; d=google.com; s=arc-20160816; b=Apq34YIl3/ER/WqbIwH7RIiU03h53KOFZV97e37RAjmGInmDg74WcHKtzlq1HeL6Db tEX8/FlpYHkcBsd2o2J8Vip0tTJRJiviC9RnA5b0NCx5JTl2K0501APu/i4/4ipj2gTQ ti3c9s7OFnaiEdyiZQ8pxnoXUCt93q5DFuMfdThxK9009m5dlCyaHn3qD04+T2PTaA+s rdCHvngsLil7O0zJ/0SFWbRNg7J50PWz+9UIr9sjIsFIMYQCZLdcL/+OTeQxA38JyOBk GWTP5VGoKuMMmvTkFY6KvORpp87dQjOoboc5DlBTqgB5oEV/AgZNgcnYpynsJqH4b+CM FXGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=s26j7V3iEZr//55S8rYqaobNc+2iO706C6wxuCttLf0=; b=el4DAqbSfFx6h0y3k5X2mrnaWmk91nF10FF6fEGfyJRdg7SHrvgM0Wh+Lx1aq+Imt5 KVi21ktsnHRYJjjfe752dsnmNf5unmzrOn7LLydNBGN15Js5R8AE/WPODMQbV9tDO/NP DpmOJFI7PnskukRyZ+iVDDMsPKH89CCsGLOZrFq4QAHDBjqUys/BwIjU+vNXK2OtL+Gy h+7GfO5t65XZsDN1qpA8YL+w24yeBhV92EEO2KbtcHQr3oSXsZVnDv0r2BrxvgWCMCpv K06ZUCWxbon22kvsyr/S4YQi4V2npbGacwtswJuH0AOtdjPpySx7hxa9ddNFceKVQIss INCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=aSb1R8bq; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id o186sor1146408oib.8.2019.05.30.14.54.21 for (Google Transport Security); Thu, 30 May 2019 14:54:21 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=aSb1R8bq; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=s26j7V3iEZr//55S8rYqaobNc+2iO706C6wxuCttLf0=; b=aSb1R8bqa6Rr+SLly5lIAn6ed5UqisHx2cm6+o2zE7jy8I/m6Ufv6nvAAvA+75/NEu m/lrayI+hShBHlah07B/ZkL3lqq696x2NUjibkLhfJPBsBJBESEtYFuDzDrckyYAlDvg XXaTAzOzEL2wAXcPUQn0S6hyluZbeHnPjkSRnnT1eJeZEbbFA3q5XaMAAPOKmsUfnE+1 7UeR4D2WS4GiveHpkiF9qIUMj9VMCxpY6cjNmQwdap0RJYIve+PbJK2N2DZsXq9TKApw jm/vr+vrW24xGM/DJ8+ArlgCGwtunsLl3XaJ3Agwt/xQPku1APV5CR+J397GTEJKbI+1 6yog== X-Google-Smtp-Source: APXvYqy/FQ8bhtfq9GgSYKUcKlSG4/DGy0PBOtr85jmyWEPil3tQV1gAaVwyI4p86YxQAPDAp9Ru0A== X-Received: by 2002:aca:5004:: with SMTP id e4mr3986999oib.179.1559253261053; Thu, 30 May 2019 14:54:21 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id v89sm1442292otb.14.2019.05.30.14.54.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:54:20 -0700 (PDT) Subject: [RFC PATCH 06/11] mm: Add membrane to free area to use as divider between treated and raw pages From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:54:18 -0700 Message-ID: <20190530215418.13974.63493.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck Add a pointer we shall call "membrane" which represents the upper boundary between the "raw" and "treated" pages. The general idea is that in order for a page to cross from one side of the membrane to the other it will need to go through the aeration treatment. By doing this we should be able to make certain that we keep the treated pages as one contiguous block within each free list. While treating the pages there may be two, but the two should merge into one before we complete the migratetype and allow it to fall back into the "settling" state. Signed-off-by: Alexander Duyck --- include/linux/mmzone.h | 38 ++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 14 ++++++++++++-- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a55fe6d2f63c..be996e8ca6b5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -87,10 +87,28 @@ static inline bool is_migrate_movable(int mt) get_pfnblock_flags_mask(page, page_to_pfn(page), \ PB_migrate_end, MIGRATETYPE_MASK) +/* + * The treatment state indicates the current state of the region pointed to + * by the treatment_mt and the membrane pointer. The general idea is that + * when we are in the "SETTLING" state the treatment area is contiguous and + * it is safe to move on to treating another migratetype. If we are in the + * "AERATING" state then the region is being actively processed and we + * would cause issues such as potentially isolating a section of raw pages + * between two sections of treated pages if we were to move onto another + * migratetype. + */ +enum treatment_state { + TREATMENT_SETTLING, + TREATMENT_AERATING, +}; + struct free_area { struct list_head free_list[MIGRATE_TYPES]; unsigned long nr_free_raw; unsigned long nr_free_treated; + struct list_head *membrane; + u8 treatment_mt; + u8 treatment_state; }; /* Used for pages not on another list */ @@ -113,6 +131,19 @@ static inline void add_to_free_area_tail(struct page *page, struct free_area *ar list_add_tail(&page->lru, &area->free_list[migratetype]); } +static inline void +add_to_free_area_treated(struct page *page, struct free_area *area, + int migratetype) +{ + area->nr_free_treated++; + + BUG_ON(area->treatment_mt != migratetype); + + /* Insert page above membrane, then move membrane to the page */ + list_add_tail(&page->lru, area->membrane); + area->membrane = &page->lru; +} + /* Used for pages which are on another list */ static inline void move_to_free_area(struct page *page, struct free_area *area, int migratetype) @@ -135,6 +166,10 @@ static inline void move_to_free_area(struct page *page, struct free_area *area, area->nr_free_raw++; } + /* push membrane back if we removed the upper boundary */ + if (area->membrane == &page->lru) + area->membrane = page->lru.next; + list_move(&page->lru, &area->free_list[migratetype]); } @@ -153,6 +188,9 @@ static inline void del_page_from_free_area(struct page *page, else area->nr_free_raw--; + if (area->membrane == &page->lru) + area->membrane = page->lru.next; + list_del(&page->lru); __ClearPageBuddy(page); __ResetPageTreated(page); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f6c067c6c784..f4a629b6af96 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -989,6 +989,11 @@ static inline void __free_one_page(struct page *page, set_page_order(page, order); area = &zone->free_area[order]; + if (PageTreated(page)) { + add_to_free_area_treated(page, area, migratetype); + return; + } + if (buddy_merge_likely(pfn, buddy_pfn, page, order) || is_shuffle_tail_page(order)) add_to_free_area_tail(page, area, migratetype); @@ -5961,8 +5966,13 @@ static void __meminit zone_init_free_lists(struct zone *zone) INIT_LIST_HEAD(&zone->free_area[order].free_list[t]); for (order = MAX_ORDER; order--; ) { - zone->free_area[order].nr_free_raw = 0; - zone->free_area[order].nr_free_treated = 0; + struct free_area *area = &zone->free_area[order]; + + area->nr_free_raw = 0; + area->nr_free_treated = 0; + area->treatment_mt = 0; + area->treatment_state = TREATMENT_SETTLING; + area->membrane = &area->free_list[0]; } } From patchwork Thu May 30 21:54:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48EC7912 for ; Thu, 30 May 2019 21:54:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3859028A12 for ; Thu, 30 May 2019 21:54:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C0E728A60; Thu, 30 May 2019 21:54:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A37528A61 for ; Thu, 30 May 2019 21:54:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B96846B0275; Thu, 30 May 2019 17:54:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B46916B0276; Thu, 30 May 2019 17:54:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E7EA6B0277; Thu, 30 May 2019 17:54:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by kanga.kvack.org (Postfix) with ESMTP id 6F6E76B0275 for ; Thu, 30 May 2019 17:54:30 -0400 (EDT) Received: by mail-ot1-f70.google.com with SMTP id q12so3469822oth.15 for ; Thu, 30 May 2019 14:54:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=ySRpFG6dIOHZ+M23tjria9Tf+GQw7/jvywjmJkXyRMo=; b=Pg5nJevJVQOS7Qv90xjEzkKAtPWV7HyH5FaRQqwK9Nnuk4YiQhgcnXCCBISalbWLrF iZRFV5E1lQzYtN8KhE4y7eUfSoDw7ZOLishvuvxCGpkX0kg1DH5cx04cD0DTxbv34ZFp 9+kVQqJGvNE128lxIGVJKdWdVPsNL6sQCgiPulx+qVVHA1UDb017YsPkek+dZeODQlYV JOGJ6RFn40Lv+VGpsrL6rRiksHLOhda5m+Py+uEWIpzS+k3qb51Uf9MB4A5JYJf04F0D oSOlbIc1bLD6ST0MsJgxfEl+7HtAZ9HhPWiWBs2xATIDLTTZnXfMBi/JLDhE/kisD/io +MHg== X-Gm-Message-State: APjAAAW5oF0dyOZmgiWiFrP2WhY12pmfmn/WC7N5Xzbfirvi21w8rpK8 UNdqs20v5kNBdMFo/QJUnCeSCG/cjLxY2kagP7koGAgv67IiqDXhYOUjEw3Zc2DiCScw5fiUkFy xhmAnIY+2qplk5YFb8XQQ9EbH5oxeLtte/mYmy9n8xMV7uATrRjv9zMwFcCehzTAo7g== X-Received: by 2002:aca:c7d7:: with SMTP id x206mr2820268oif.92.1559253270117; Thu, 30 May 2019 14:54:30 -0700 (PDT) X-Received: by 2002:aca:c7d7:: with SMTP id x206mr2820230oif.92.1559253269030; Thu, 30 May 2019 14:54:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253269; cv=none; d=google.com; s=arc-20160816; b=Hn7PLOBTPXfRTLjiqNTm6qaZREFAzmOLiV8Tq14Y6sOCXboZytpYEhjwfzOhCs1Bgy +xW1/g376a3gMLxhbaSoon6UD7tfPOez6aH90ifZ21B0qNqRAFxio1Z9oTJsM5VS6uPe N5sTRe6QOwQC+WJE2QG1n7Jx9B+SqDccPyPvNJHj76x//NVN165l1KnHyEP6R/FOU3wJ 1NJYsu11G26ksZj2wzxEChz38YwZKpZzRuB0mUIuhJuLs8QCZGZzKkIMSIEIOhVH+SV9 aGlra2Cpd4Jlq6YkFMOFQpSVNC1SKgOiDSmpUIJpy5YoixbMpl+MQI0Zkkph8yDJ7jse zRQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=ySRpFG6dIOHZ+M23tjria9Tf+GQw7/jvywjmJkXyRMo=; b=WqBIIMf48PF1Hvh9OP6GbcyWTLzEjbyky3H4xebJNbxVfAGhySJjMsCajIPjk7FLtE oXW/dClHl5HVHY8EhBvPUDZh95BNQxphx4b82JCTRYycGTcgz+WfBJMZqjQ101TJGEms j+WXfuY2pZJRuFwHyHn38UZdwJQRDJl1RAF7ExHw7YjpuyR0y1R1u2bgll793PnYq5L4 Dw0yjYTtKx2tOKE9sxKh5JmuIvKNGhzVDo4RdrP+Ay4lvxytPdvaoCF+cJFbilUk0U5s 61YEfyTJ5kVNQdxd3SpaCyWjk2Wq/46s84+cXXL07Eb/SratEy1BEMtNGeY1wO2VKPkt ythg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rcau95AX; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id f6sor1931288otb.115.2019.05.30.14.54.28 for (Google Transport Security); Thu, 30 May 2019 14:54:29 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rcau95AX; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=ySRpFG6dIOHZ+M23tjria9Tf+GQw7/jvywjmJkXyRMo=; b=rcau95AXvxEykUxo3o4rkucpfbR1LX1x5X5CXkMhzd1fqmFgZZwKoTCbMpKKbYqFfC UbdBILZlqxd2RuRIYCEGRxN9ep9jYLYhRreEWra8LWhyYsMzByoobPErsAneYypuf6Gu PE0cjm4SCxwhQ+NB4y+KyiiEoBG9ckxFVgWYm/GtS2/Qw+K5GSC6c6a8Nvh8Z/zxoIGG Y4buUb8l/83NXSsb2o15MmHUsRTwTI8cOyyoiVqfkSpG+HVP3nQTtHm+7oCsJFRGsagc ei5rEo+G+E2/wyIldOG821aJC1RHYMP9oREGt0ATmV6jeHMyQNAdMUGC3sbUmsIg0r9/ beYw== X-Google-Smtp-Source: APXvYqxrTapW1llgCNvcp1SuLkSKr6HRUWIeylHL8O4HoCla8qYP0tDZcevscz28QJb95+UxLu1Szg== X-Received: by 2002:a9d:3285:: with SMTP id u5mr4625825otb.266.1559253268401; Thu, 30 May 2019 14:54:28 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id p63sm866455oih.1.2019.05.30.14.54.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:54:27 -0700 (PDT) Subject: [RFC PATCH 07/11] mm: Add support for acquiring first free "raw" or "untreated" page in zone From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:54:26 -0700 Message-ID: <20190530215426.13974.82813.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck In order to be able to "treat" memory in an asynchonous fashion we need a way to acquire a block of memory that isn't already treated, and then flush that back in a way that we will not pick it back up again. To achieve that this patch adds a pair of functions. One to fill a list with pages to be treated, and another that will flush out the list back to the buddy allocator. Signed-off-by: Alexander Duyck --- include/linux/gfp.h | 6 +++ mm/page_alloc.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 113 insertions(+) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index fb07b503dc45..407a089d861f 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -559,6 +559,12 @@ extern void *page_frag_alloc(struct page_frag_cache *nc, void drain_all_pages(struct zone *zone); void drain_local_pages(struct zone *zone); +#ifdef CONFIG_AERATION +struct page *get_raw_pages(struct zone *zone, unsigned int order, + int migratetype); +void free_treated_page(struct page *page); +#endif + void page_alloc_init_late(void); /* diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f4a629b6af96..e79c65413dc9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2155,6 +2155,113 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, return NULL; } +#ifdef CONFIG_AERATION +static struct page *get_raw_page_from_free_area(struct free_area *area, + int migratetype) +{ + struct list_head *head = &area->free_list[migratetype]; + struct page *page; + + /* If we have not worked in this free_list before reset membrane */ + if (area->treatment_mt != migratetype) { + area->treatment_mt = migratetype; + area->membrane = head; + } + + /* Try to pulling in any untreated pages above the the membrane */ + page = list_last_entry(area->membrane, struct page, lru); + list_for_each_entry_from_reverse(page, head, lru) { + /* + * If the page in front of the membrane is treated then try + * skimming the top to see if we have any untreated pages + * up there. + */ + if (PageTreated(page)) { + page = list_first_entry(head, struct page, lru); + if (PageTreated(page)) + break; + } + + /* update state of treatment */ + area->treatment_state = TREATMENT_AERATING; + + return page; + } + + /* + * At this point there are no longer any untreated pages between + * the membrane and the first entry of the list. So we can safely + * set the membrane to the top of the treated region and will mark + * the current migratetype as complete for now. + */ + area->membrane = &page->lru; + area->treatment_state = TREATMENT_SETTLING; + + return NULL; +} + +/** + * get_raw_pages - Provide a "raw" page for treatment by the aerator + * @zone: Zone to draw pages from + * @order: Order to draw pages from + * @migratetype: Migratetype to draw pages from + * + * This function will obtain a page that does not have the Treated value + * set in the page type field. It will attempt to fetch a "raw" page from + * just above the "membrane" and if that is not available it will attempt + * to pull a "raw" page from the head of the free list. + * + * The page will have the migrate type and order stored in the page + * metadata. + * + * Return: page pointer if raw page found, otherwise NULL + */ +struct page *get_raw_pages(struct zone *zone, unsigned int order, + int migratetype) +{ + struct free_area *area = &(zone->free_area[order]); + struct page *page; + + /* Find a page of the appropriate size in the preferred list */ + page = get_raw_page_from_free_area(area, migratetype); + if (page) { + del_page_from_free_area(page, area); + + /* record migratetype and order within page */ + set_pcppage_migratetype(page, migratetype); + set_page_private(page, order); + __mod_zone_freepage_state(zone, -(1 << order), migratetype); + } + + return page; +} +EXPORT_SYMBOL_GPL(get_raw_pages); + +/** + * free_treated_page - Return a now-treated "raw" page back where we got it + * @page: Previously "raw" page that can now be returned after treatment + * + * This function will pull the zone, migratetype, and order information out + * of the page and attempt to return it where it found it. We default to + * using free_one_page to return the page as it is possible that the + * pageblock might have been switched to an isolate migratetype during + * treatment. + */ +void free_treated_page(struct page *page) +{ + unsigned int order, mt; + struct zone *zone; + + zone = page_zone(page); + mt = get_pcppage_migratetype(page); + order = page_private(page); + + set_page_private(page, 0); + + free_one_page(zone, page, page_to_pfn(page), order, mt); +} +EXPORT_SYMBOL_GPL(free_treated_page); +#endif /* CONFIG_AERATION */ /* * This array describes the order lists are fallen back to when From patchwork Thu May 30 21:54:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969291 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3D7E414C0 for ; Thu, 30 May 2019 21:54:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2B1FC28A61 for ; Thu, 30 May 2019 21:54:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1F2AD28A6A; Thu, 30 May 2019 21:54:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0370728904 for ; Thu, 30 May 2019 21:54:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15E1B6B0277; Thu, 30 May 2019 17:54:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 110006B0278; Thu, 30 May 2019 17:54:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF0E76B0279; Thu, 30 May 2019 17:54:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by kanga.kvack.org (Postfix) with ESMTP id BFAF06B0277 for ; Thu, 30 May 2019 17:54:38 -0400 (EDT) Received: by mail-ot1-f70.google.com with SMTP id a21so3469625otl.14 for ; Thu, 30 May 2019 14:54:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=VrUdZuuah1ePBnYB1FF7mjrwNfzoFpJZPyHrGDL/mrg=; b=hMhg7lgJWdHRbYFUOzx2HipJ7S5BXndZT6sCyJjorxM49FS01Qc9BQ86yB9w4Y5HcL R3CbkjDwzyFBN34f5mNdy8A15lLSRRotVU0rR512R1M6cXTqR7UM0n9qbnIVeaWOfRIW rAXyTiG89pIzSyrcgqPbg3odoi5E5NiR7bs8ZXGFr2ZihtV5SuUKZ6mIhaniVYEFETqQ 8m5hdpWCkX6J88Vqv62JcqXtlfTHBqBP+gynugjshJG6MHQ4Kj4ZaYGE6pHl4JNKz8xd PSWo+kHpmvnFCNhd2cfTFeJl35puUast/ZobsdjKmCUVgSuVsuFeNp5ZHHN8u7A+n3JK l6Xg== X-Gm-Message-State: APjAAAWL9+2A12XDA2kdj2RySu05Wv2BGjzOTzNG/8e8LqjVMzq8guvL ctbzqHHfFFrzFlUhf3dH4irMhZ0P5XQ+m8xX15KYAn565Yrf8v+lSRICO9XqV4iO7SZT7gah6f0 tgKxcoyagOMJLMNBQ8dUCPErmnUlJH3e8iUGpi3vRKvg8PlOhrZquy2ix+pccIsbXtg== X-Received: by 2002:aca:cd15:: with SMTP id d21mr3896088oig.41.1559253278409; Thu, 30 May 2019 14:54:38 -0700 (PDT) X-Received: by 2002:aca:cd15:: with SMTP id d21mr3896052oig.41.1559253277142; Thu, 30 May 2019 14:54:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253277; cv=none; d=google.com; s=arc-20160816; b=BGY6YdkSUBFwEcIphoX54Hqjvy9+mWMOtMfm57t0uViCBjtbRcn9k11I7KXzuX27gE A64nTFuZHdfIle/k4rBDNdTnlDjESVTQOkHSfz0+cWGpTYK2UG4QKPqz2fxICIszVkVk NKaKi0kS+d2r1ZmMqHdXhgT6c+jtJwi0iKxBVUVD184sOEJsoVoj+lsUNxVJeC48U4LJ ChJnhfiWZarOqSxHVSWHdNZU4PfPV+7sOIYiwGajk4Kx1egSUcvkYwnbgkBrUhlxbS4M FxbHuwKApe3nQkiNhB30flVyfBkmWC90+gNV/wymGOHTWpSylYrjXgGHc5gYfheBL/q8 6rUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=VrUdZuuah1ePBnYB1FF7mjrwNfzoFpJZPyHrGDL/mrg=; b=b97rRlAjPk9TcmqDVRxtuWyCnyPPNap1MHSlI+EbX9ak6iLOSJPt/z5TfoxyW1a8YZ 2diS8J9B8G3Yk1Mx7cCd905v/wpf4/0jUSKRuDFXFYEv1KPCaxazARf0TUKmOrDd6Ngm 4mdwUdqKi0aekHGD9b7enANU9hAvdR6WiNoeEURJRLdVDQ7aTmj+CZGUpwrPeOmZWxPM TeDZwZd74MfKIA4nfZW252cNywExIEifrb89WLte3AcAjY3HtWM319v6MV6MlO9Fhe+u C/K6AwLG76biaInYVyxxHqQbk6nzET8YxD99etEWb8c2qLDOmfX3deb2Fsd0MTvaRIXI +Cqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=RWkM8Yl5; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id y4sor1912650otq.175.2019.05.30.14.54.37 for (Google Transport Security); Thu, 30 May 2019 14:54:37 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=RWkM8Yl5; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=VrUdZuuah1ePBnYB1FF7mjrwNfzoFpJZPyHrGDL/mrg=; b=RWkM8Yl5fLEz/8vjf5jVcRJXT8EF4ayMTXv7VUaV9qHT5iuiRJFkfD2jAc7TX5vZE9 R6fwxU1uo7dbhu+Ta23F46YEx+EpstmpbboqE8ZzWfc73qPEBl8SVwYR3OxGein+w+pF 4TKspLHWCWLJrXCjZkUbFlNZ2p32JYAHCBjCKjE0s7lDIugHB2trrFgc3MpeSVSn9ypB iv0ZOemcgSITjKPMhkRw5HzR45ku6p6dVQrLPKbpZ6koXhcRFmuK2wVawtM0hYmcTq/i rJ860wczu4at7pY2GoVb3tGpoh2NxTlvLYvJS5tmmmm2w6Kv/KKg9NscA0k+VbbULgUG PZgQ== X-Google-Smtp-Source: APXvYqw7g1L+G0cqINE4PJyWCvfS0dKJpUSfVKFCiCqnvg3eJwWRViAbuOlMVDgf+SLDserE4hl5Ig== X-Received: by 2002:a05:6830:1318:: with SMTP id p24mr2104325otq.75.1559253276059; Thu, 30 May 2019 14:54:36 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id z16sm1117798ote.50.2019.05.30.14.54.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:54:35 -0700 (PDT) Subject: [RFC PATCH 08/11] mm: Add support for creating memory aeration From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:54:33 -0700 Message-ID: <20190530215433.13974.43219.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck Add support for "aerating" memory in a guest by pushing individual pages out. This patch is meant to add generic support for this by adding a common framework that can be used later by drivers such as virtio-balloon. Signed-off-by: Alexander Duyck --- include/linux/memory_aeration.h | 54 +++++++ mm/Kconfig | 5 + mm/Makefile | 1 mm/aeration.c | 320 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 380 insertions(+) create mode 100644 include/linux/memory_aeration.h create mode 100644 mm/aeration.c diff --git a/include/linux/memory_aeration.h b/include/linux/memory_aeration.h new file mode 100644 index 000000000000..5ba0e634f240 --- /dev/null +++ b/include/linux/memory_aeration.h @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MEMORY_AERATION_H +#define _LINUX_MEMORY_AERATION_H + +#include +#include +#include + +struct zone; + +#define AERATOR_MIN_ORDER pageblock_order + +struct aerator_dev_info { + unsigned long capacity; + struct list_head batch_reactor; + atomic_t refcnt; + void (*react)(struct aerator_dev_info *a_dev_info); +}; + +extern struct static_key aerator_notify_enabled; + +void aerator_cycle(void); +void __aerator_notify(struct zone *zone, int order); + +/** + * aerator_notify_free - Free page notification that will start page processing + * @page: Last page processed + * @zone: Pointer to current zone of last page processed + * @order: Order of last page added to zone + * + * This function is meant to act as a screener for __aerator_notify which + * will determine if a give zone has crossed over the high-water mark that + * will justify us beginning page treatment. If we have crossed that + * threshold then it will start the process of pulling some pages and + * placing them in the batch_reactor list for treatment. + */ +static inline void +aerator_notify_free(struct page *page, struct zone *zone, int order) +{ + if (!static_key_false(&aerator_notify_enabled)) + return; + + if (order < AERATOR_MIN_ORDER) + return; + + __aerator_notify(zone, order); +} + +void aerator_shutdown(void); +int aerator_startup(struct aerator_dev_info *sdev); + +#define AERATOR_ZONE_BITS (BITS_TO_LONGS(MAX_NR_ZONES) * BITS_PER_LONG) +#define AERATOR_HWM_BITS (AERATOR_ZONE_BITS * MAX_NUMNODES) +#endif /*_LINUX_MEMORY_AERATION_H */ diff --git a/mm/Kconfig b/mm/Kconfig index f0c76ba47695..34680214cefa 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -236,6 +236,11 @@ config COMPACTION linux-mm@kvack.org. # +# support for memory aeration +config AERATION + bool + +# # support for page migration # config MIGRATION diff --git a/mm/Makefile b/mm/Makefile index ac5e5ba78874..26c2fcd2b89d 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -104,3 +104,4 @@ obj-$(CONFIG_HARDENED_USERCOPY) += usercopy.o obj-$(CONFIG_PERCPU_STATS) += percpu-stats.o obj-$(CONFIG_HMM) += hmm.o obj-$(CONFIG_MEMFD_CREATE) += memfd.o +obj-$(CONFIG_AERATION) += aeration.o diff --git a/mm/aeration.c b/mm/aeration.c new file mode 100644 index 000000000000..aaf8af8d822f --- /dev/null +++ b/mm/aeration.c @@ -0,0 +1,320 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include + +static unsigned long *aerator_hwm; +static struct aerator_dev_info *a_dev_info; +struct static_key aerator_notify_enabled; + +void aerator_shutdown(void) +{ + static_key_slow_dec(&aerator_notify_enabled); + + while (atomic_read(&a_dev_info->refcnt)) + msleep(20); + + kfree(aerator_hwm); + aerator_hwm = NULL; + + a_dev_info = NULL; +} +EXPORT_SYMBOL_GPL(aerator_shutdown); + +int aerator_startup(struct aerator_dev_info *sdev) +{ + size_t size = BITS_TO_LONGS(AERATOR_HWM_BITS) * sizeof(unsigned long); + unsigned long *hwm; + + if (a_dev_info || aerator_hwm) + return -EBUSY; + + a_dev_info = sdev; + + atomic_set(&sdev->refcnt, 0); + + hwm = kzalloc(size, GFP_KERNEL); + if (!hwm) { + aerator_shutdown(); + return -ENOMEM; + } + + aerator_hwm = hwm; + + static_key_slow_inc(&aerator_notify_enabled); + + return 0; +} +EXPORT_SYMBOL_GPL(aerator_startup); + +static inline unsigned long *get_aerator_hwm(int nid) +{ + if (!aerator_hwm) + return NULL; + + return aerator_hwm + (BITS_TO_LONGS(MAX_NR_ZONES) * nid); +} + +static int __aerator_fill(struct zone *zone, unsigned int size) +{ + struct list_head *batch = &a_dev_info->batch_reactor; + unsigned long nr_raw = 0; + unsigned int len = 0; + unsigned int order; + + for (order = MAX_ORDER; order-- != AERATOR_MIN_ORDER;) { + struct free_area *area = &(zone->free_area[order]); + int mt = area->treatment_mt; + + /* + * If there are no untreated pages to pull + * then we might as well skip the area. + */ + while (area->nr_free_raw) { + unsigned int count = 0; + struct page *page; + + /* + * If we completed aeration we can let the current + * free list work on settling so that a batch of + * new raw pages can build. In the meantime move on + * to the next migratetype. + */ + if (++mt >= MIGRATE_TYPES) + mt = 0; + + /* + * Pull pages from free list until we have drained + * it or we have filled the batch reactor. + */ + while ((page = get_raw_pages(zone, order, mt))) { + list_add(&page->lru, batch); + + if (++count == (size - len)) + return size; + } + + /* + * If we pulled any pages from this migratetype then + * we must move on to a new free area as we cannot + * move the membrane until after we have decanted the + * pages currently being aerated. + */ + if (count) { + len += count; + break; + } + } + + /* + * Keep a running total of the raw packets we have left + * behind. We will use this to determine if we should + * clear the HWM flag. + */ + nr_raw += area->nr_free_raw; + } + + /* + * If there are no longer enough free pages to fully populate + * the aerator, then we can just shut it down for this zone. + */ + if (nr_raw < a_dev_info->capacity) { + unsigned long *hwm = get_aerator_hwm(zone_to_nid(zone)); + + clear_bit(zone_idx(zone), hwm); + atomic_dec(&a_dev_info->refcnt); + } + + return len; +} + +static unsigned int aerator_fill(int nid, int zid, int budget) +{ + pg_data_t *pgdat = NODE_DATA(nid); + struct zone *zone = &pgdat->node_zones[zid]; + unsigned long flags; + int len; + + spin_lock_irqsave(&zone->lock, flags); + + /* fill aerator with "raw" pages */ + len = __aerator_fill(zone, budget); + + spin_unlock_irqrestore(&zone->lock, flags); + + return len; +} + +static void aerator_fill_and_react(void) +{ + int budget = a_dev_info->capacity; + int nr; + + /* + * We should never be calling this function while there are already + * pages in the reactor being aerated. If we are called under such + * a circumstance report an error. + */ + BUG_ON(!list_empty(&a_dev_info->batch_reactor)); +retry: + /* + * We want to hold one additional reference against the number of + * active hints as we may clear the hint that originally brought us + * here. We will clear it after we have either vaporized the content + * of the pages, or if we discover all pages were stolen out from + * under us. + */ + atomic_inc(&a_dev_info->refcnt); + + for_each_set_bit(nr, aerator_hwm, AERATOR_HWM_BITS) { + int node_id = nr / AERATOR_ZONE_BITS; + int zone_id = nr % AERATOR_ZONE_BITS; + + budget -= aerator_fill(node_id, zone_id, budget); + if (!budget) + goto start_aerating; + } + + if (unlikely(list_empty(&a_dev_info->batch_reactor))) { + /* + * If we never generated any pages, and we were holding the + * only remaining reference to active hints then we can + * just let this go for now and go idle. + */ + if (atomic_dec_and_test(&a_dev_info->refcnt)) + return; + + /* + * There must be a bit populated somewhere, try going + * back through and finding it. + */ + goto retry; + } + +start_aerating: + a_dev_info->react(a_dev_info); +} + +void aerator_decant(void) +{ + struct list_head *list = &a_dev_info->batch_reactor; + struct page *page; + + /* + * This function should never be called on an empty list. If so it + * points to a bug as we should never be running the aerator when + * the list is empty. + */ + WARN_ON(list_empty(&a_dev_info->batch_reactor)); + + while ((page = list_first_entry_or_null(list, struct page, lru))) { + list_del(&page->lru); + + __SetPageTreated(page); + + free_treated_page(page); + } +} + +/** + * aerator_cycle - drain, fill, and start aerating another batch of pages + * + * This function is at the heart of the aerator. It should be called after + * the previous batch of pages has finished being processed by the aerator. + * It will drain the aerator, refill it, and start the next set of pages + * being processed. + */ +void aerator_cycle(void) +{ + aerator_decant(); + + /* + * Now that the pages have been flushed we can drop our reference to + * the active hints list. If there are no further hints that need to + * be processed we can simply go idle. + */ + if (atomic_dec_and_test(&a_dev_info->refcnt)) + return; + + aerator_fill_and_react(); +} +EXPORT_SYMBOL_GPL(aerator_cycle); + +static void __aerator_fill_and_react(struct zone *zone) +{ + /* + * We should never be calling this function while there are already + * pages in the list being aerated. If we are called under such a + * circumstance report an error. + */ + BUG_ON(!list_empty(&a_dev_info->batch_reactor)); + + /* + * We want to hold one additional reference against the number of + * active hints as we may clear the hint that originally brought us + * here. We will clear it after we have either vaporized the content + * of the pages, or if we discover all pages were stolen out from + * under us. + */ + atomic_inc(&a_dev_info->refcnt); + + __aerator_fill(zone, a_dev_info->capacity); + + if (unlikely(list_empty(&a_dev_info->batch_reactor))) { + /* + * If we never generated any pages, and we were holding the + * only remaining reference to active hints then we can just + * let this go for now and go idle. + */ + if (atomic_dec_and_test(&a_dev_info->refcnt)) + return; + + /* + * Another zone must have populated some raw pages that + * need to be processed. Release the zone lock and process + * that zone instead. + */ + spin_unlock(&zone->lock); + aerator_fill_and_react(); + } else { + /* Release the zone lock and begin the page aerator */ + spin_unlock(&zone->lock); + a_dev_info->react(a_dev_info); + } + + /* Reaquire lock so we can resume processing this zone */ + spin_lock(&zone->lock); +} + +void __aerator_notify(struct zone *zone, int order) +{ + int node_id = zone_to_nid(zone); + int zone_id = zone_idx(zone); + unsigned long *hwm; + + if (zone->free_area[order].nr_free_raw < (2 * a_dev_info->capacity)) + return; + + hwm = get_aerator_hwm(node_id); + + /* + * We an use separate test and set operations here as there + * is nothing else that can set or clear this bit while we are + * holding the zone lock. The advantage to doing it this way is + * that we don't have to dirty the cacheline unless we are + * changing the value. + */ + if (test_bit(zone_id, hwm)) + return; + set_bit(zone_id, hwm); + + if (atomic_fetch_inc(&a_dev_info->refcnt)) + return; + + __aerator_fill_and_react(zone); +} +EXPORT_SYMBOL_GPL(__aerator_notify); + From patchwork Thu May 30 21:54:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 24EB814C0 for ; Thu, 30 May 2019 21:54:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14C2628A5B for ; Thu, 30 May 2019 21:54:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0922628C45; Thu, 30 May 2019 21:54:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79B5328B01 for ; Thu, 30 May 2019 21:54:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 136ED6B0279; Thu, 30 May 2019 17:54:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0E70B6B027A; Thu, 30 May 2019 17:54:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA3646B027B; Thu, 30 May 2019 17:54:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by kanga.kvack.org (Postfix) with ESMTP id C03C16B0279 for ; Thu, 30 May 2019 17:54:44 -0400 (EDT) Received: by mail-ot1-f69.google.com with SMTP id t17so3462889otp.19 for ; Thu, 30 May 2019 14:54:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=+Hql3UnNz+U+1lYmmCsad2oNlY6bZIgkjk9Ecr29j+A=; b=UxTgNbaG6c3MpSW6k6/zlGPKJQi7X9ZgytTCnz8H4F9nwhPA3YcC/y25BrCxYJZOAX 0/mL64s/5lfORb5CIaG3WSV0rG4T6RqoJyUBNmXjUsUaARIVdVCnazIOV3u8Wg4lqlcD 5tWmMoOGFR4Fn3l0+sEqxIyZIKfuxx4a/jR1p13/xQiB54FfqKCN02MXna/MGZlqGvW6 ScdKPpj6Ndv+1NYN0rQG7p2447ZzmNng2Tv58kIM9v1IjBMluLKtfdIpTK4LK6IftQNB 84ARbnzv3f2OpUWIOF7K8kVdbX5ufEPVdB/oFb5A8cInnnFOO3kcFW/+nrEK2Y3bFZdo iknA== X-Gm-Message-State: APjAAAV5GsrN3HZY4/doCLukiVvCup2Xx8UTAJXPLsTkegrhq4No/Wr/ l+M6EVm7vpv5GxmI+Kq0uAzZJXP4hHmTWSDhPXz8hjBrgE5VZOGNSoNtCPbr9ynnxFgZawPgKYs G2l1sq02J9MRorSRbFNI0GCJaGUgwmPj9JRZ5fTzlyq8J74R1gFoNVfIWvRCB6x+Y7g== X-Received: by 2002:aca:c3d7:: with SMTP id t206mr60926oif.129.1559253284466; Thu, 30 May 2019 14:54:44 -0700 (PDT) X-Received: by 2002:aca:c3d7:: with SMTP id t206mr60902oif.129.1559253283651; Thu, 30 May 2019 14:54:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253283; cv=none; d=google.com; s=arc-20160816; b=oT9xwrBHh1ve2sRqVB4Gxl8Ndcwwn7dA0PqYWtVZxilxr6505hU9YgmmgLuz7Moxxa 1m+HNS2B2isJ3TQ35U6EZtvlvUsqIFbpUTOhRE4Ve1CR5f4t4wGGdsAHXUmZ9LYgDM5n Vbhg0w1jHoHTcX5Pe8hB3tH3jw+M/CGgtjVGx4ou72Ei2SapWE0ZGhrTHkB/OZqYsJ4d 8FNO/p+4wSveCuGPk1PokqxbIF6uzS0y4NAr1dAsdPkXI3i6SWLnmHpUScqKcwRQ2u9a 2rWwZsTew2MMSFrCy/4UnhEMlxsUYsb00RQQDkRZM3HlSkHP/rkPsAtpJ6ofqedMEmez 5Xfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=+Hql3UnNz+U+1lYmmCsad2oNlY6bZIgkjk9Ecr29j+A=; b=Ney+cVNEeykP78urVzQUFaYyIE3vshxrJsEhkjuRHfDMzx5FKT+MIRHtPA2z+xx6Fr SwlNW3HDwZVvhtOuDR2FkEgXscSOURUpCuOaKyH9j90Z3pUoShzP1orbG6jRZwZ/+85q E0I5mqlgn4w7EW4/KBycvYvO64ekh+eE4WqDTzp5ZVeNNoD2FYxpHktSsPbhOiv3ch4S bM5s17Sns+y9fkw6tUYEU57Jf0GMwJenSS6i0WCEfmTHMORIiwBml+q5e8EemRWDi/ke sSeBjtyklRlCYGF1Ie7qsaCgEjccEgdLNE86/A3tZISgXTwqWUEVRrTyKVBCePfzlZnc 9G3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=BXs8yL23; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x6sor1968077otp.81.2019.05.30.14.54.43 for (Google Transport Security); Thu, 30 May 2019 14:54:43 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=BXs8yL23; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=+Hql3UnNz+U+1lYmmCsad2oNlY6bZIgkjk9Ecr29j+A=; b=BXs8yL238Nx++nVmRKlg3dNF8CTnKOpeTXH0AG07gUebKWVwBReMaUvxW+ADvzwJwk HL7Z2/iu94p7I4zy38LSM+GFa9I3DNkDzScHJPMvIDd3EIXwnwx7TU9/jRgQ2bPHCNTN b2ZeQhQnwYYnXtsyLbJi6ZBBU7WMzu+BduNJjFj7Yb342UxFjbViJCZZ1aWCPfh3p5z3 UqJZVkfskln9C5j/67ONPb9LHHYvE0bD2NGQUnSUtNRASvc24zb66/hAutak8OFQ70Mi +a9a/wdBysp+q2E181m0otXqpxVwLVHU5jpjDa7eHqaV2QssIfLYiRjA53oBXVcPZziH Oy6Q== X-Google-Smtp-Source: APXvYqxGTR2Bptz8i/SgWXoKTL+4Bu+dJLGidAXFm62PcUBVrIR9eyVW68+tWL2IUSUeElRSKJsvlg== X-Received: by 2002:a9d:5ec:: with SMTP id 99mr4311813otd.57.1559253283263; Thu, 30 May 2019 14:54:43 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id n7sm1450349oih.18.2019.05.30.14.54.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:54:42 -0700 (PDT) Subject: [RFC PATCH 09/11] mm: Count isolated pages as "treated" From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:54:41 -0700 Message-ID: <20190530215441.13974.33609.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck Treat isolated pages as though they have already been treated. We do this so that we can avoid trying to treat pages that have been marked for isolation. The issue is that we don't want to run into issues where we are treating a page, and when we put it back we find it has been moved into the isolated migratetype, nor would we want to pull pages out of the isolated migratetype and then find that they are now being located in a different migratetype. To avoid those issues we can specifically mark all isolated pages as being "treated" and avoid special case handling for them since they will never be merged anyway, so we can just add them to the head of the free_list. In addition we will skip over the isolate migratetype when getting raw pages. Signed-off-by: Alexander Duyck --- include/linux/mmzone.h | 7 +++++++ mm/aeration.c | 8 ++++++-- mm/page_alloc.c | 2 +- 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index be996e8ca6b5..f749ccfcc62a 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -137,6 +137,13 @@ static inline void add_to_free_area_tail(struct page *page, struct free_area *ar { area->nr_free_treated++; +#ifdef CONFIG_MEMORY_ISOLATION + /* Bypass membrane for isolated pages, all are considered "treated" */ + if (migratetype == MIGRATE_ISOLATE) { + list_add(&page->lru, &area->free_list[migratetype]); + return; + } +#endif BUG_ON(area->treatment_mt != migratetype); /* Insert page above membrane, then move membrane to the page */ diff --git a/mm/aeration.c b/mm/aeration.c index aaf8af8d822f..f921295ed3ae 100644 --- a/mm/aeration.c +++ b/mm/aeration.c @@ -1,6 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include +#include #include #include #include @@ -83,8 +85,10 @@ static int __aerator_fill(struct zone *zone, unsigned int size) * new raw pages can build. In the meantime move on * to the next migratetype. */ - if (++mt >= MIGRATE_TYPES) - mt = 0; + do { + if (++mt >= MIGRATE_TYPES) + mt = 0; + } while (is_migrate_isolate(mt)); /* * Pull pages from free list until we have drained diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e79c65413dc9..e3800221414b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -989,7 +989,7 @@ static inline void __free_one_page(struct page *page, set_page_order(page, order); area = &zone->free_area[order]; - if (PageTreated(page)) { + if (is_migrate_isolate(migratetype) || PageTreated(page)) { add_to_free_area_treated(page, area, migratetype); return; } From patchwork Thu May 30 21:54:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969301 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA03F912 for ; Thu, 30 May 2019 21:54:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE4C728979 for ; Thu, 30 May 2019 21:54:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D2A7B28AD3; Thu, 30 May 2019 21:54:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3B7028C53 for ; Thu, 30 May 2019 21:54:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CA9E6B027B; Thu, 30 May 2019 17:54:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 57ACC6B027C; Thu, 30 May 2019 17:54:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4427C6B027D; Thu, 30 May 2019 17:54:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by kanga.kvack.org (Postfix) with ESMTP id 15F976B027B for ; Thu, 30 May 2019 17:54:52 -0400 (EDT) Received: by mail-oi1-f197.google.com with SMTP id l12so2762413oii.10 for ; Thu, 30 May 2019 14:54:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=/mUEYZKWYYcL2BHkb0DxPuaatZyzvoF9cSukuFqsj94=; b=nazV7d+winARXY+0pE/IdIoMmKZHP5OEYZNw9qx7Vk7Ebezy/taqwguezmujH2WpNS /B6t94vZFUgWdKLa3/nQ35BudOVRFscnbWizSdkRJUTTOO35J/IaNrmu8XFk1pHy0xpH eZbUIL2ofjiti3tiChn7tMYV3017q7BOhhi/GpzEDhEHTco4omELhHQVekFF0IXCmDoW Cd0iThBd54IojCK1+dCIIa5lRvIbcbGMrA0oCLLPmX+XyTMNZC1J0rgOcsPnjsKlWCLj ds4BUurrn3qDbcOAtjyyJtSRZHEcq1mD96vpgwe5i1dlwYaR3GJcNrITotMcQxYw/Gbl IuHw== X-Gm-Message-State: APjAAAVjqDeZjRSrcnOEfH2mr3/9Yavs4qa8h/EwPssvtqISXKYfHmV/ wmNYT6ZcZLYp0OG8aJyHDqlgoHneh8oa5s4RraCV0VsmEqeexMaERZVqFcP3saUCXsfz6uXJVB/ 1ol5UeJ87za9RnaDCLJExjaTAy5EKUVPTi2ldIb+6hilXKmHBrCMiYo+SkLLH4ePIxA== X-Received: by 2002:a05:6830:209a:: with SMTP id y26mr4514099otq.232.1559253291777; Thu, 30 May 2019 14:54:51 -0700 (PDT) X-Received: by 2002:a05:6830:209a:: with SMTP id y26mr4514055otq.232.1559253290981; Thu, 30 May 2019 14:54:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253290; cv=none; d=google.com; s=arc-20160816; b=IJcN+JAWy7DjKC4V3pwTMnA0pmBSmYxnoLLap7Khxc1eKSOpl0OYgPNY9V2FyAiIMH Q1ahMc8/TZrDW2PhuuYEAdQYigaKJxVeD7yLt03rOTt01ThZoPzLEt4asRG+w6E6Nb+Q zghWRXSy1bcFsLwKgJRfL3r9Hpr1WzMsVU4drJqMO+3zVU08yMoZjp4wSnje0S+dnSM7 2udTyjVulp4FlMu2/YsSvSRVkLjln7yPXeHFcDkfFmnu07BRkUW97fC8u7r9sJ6Vpp2V M4MtXj+gzPPXT9ACgtpkNa1rW/oDcybsy/218GN38DjQB6rn7EH4+GZqN6m6Uvh3wnwJ HUzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=/mUEYZKWYYcL2BHkb0DxPuaatZyzvoF9cSukuFqsj94=; b=NjyUFFEo4S2qPqlv/51+QZcWR0nF2sdnYfT+92of0pHW+bmfe/cCh6dCrgDH7swZUj ld7PCSKVdEeHOLO+9LLmplyZYkfo5vJyNocbEpCAP7klRKR4oz7b2aVyT38eQ9zNExGo kcAuK5bPB+zbME6+hvONTgFOXJYmkQdtKKGHLn4VU7pZz4f3dxjjIhbQOg7Zi70qfkhL FlgJyVjl4/OhXOa8E5PIdqdHIv/eGBRBU4uhBJV6Wf4uLWqHc+vtwNxsf3fEVKUCu096 Gxn/0t8zIk11lcC8Ie8D3+KuOe+h9nLJD6UKHz9t783Pli7+KyEyTgw7rbweO6TVdBb3 q9Gw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=cbR7FE1z; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p29sor1981499otf.15.2019.05.30.14.54.50 for (Google Transport Security); Thu, 30 May 2019 14:54:50 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=cbR7FE1z; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=/mUEYZKWYYcL2BHkb0DxPuaatZyzvoF9cSukuFqsj94=; b=cbR7FE1zdTHh1NeELi13nyRj1xb4yqLhmeuGlEbyivkZsLIB39JafRdCTyWZW4ufux df2TuWOVTJwY0SaSOo7mkmtztkSdvUnXcqR/MeoOGe42k0iFfT4bBFCPz0bxyURY7TUZ bhRHulEaACfjcyPUPtgEkThh7MyjbEWB2YIPTIEBYQmeZqG53q+hLORM9WmGLoudMFdL om1G4JzlXwN33wwnyKo34fnqr5EGubW3LDFIsN26NjyHtyi8wXc2VgvmiU+I8gU1ck9p 3SqhZ6xJNojk92m6bkbBN9LZm78eARhs3gL4w27+beyXfel6y7AYi/KV+k8A1EsqYxBS LlFA== X-Google-Smtp-Source: APXvYqx01mVsNW6pIQK0hwBzQcE1nW+e4GzPg6f2gyQVuJls5woMO9c9GDOi7DlASV/rWA1A7XKMvA== X-Received: by 2002:a9d:378b:: with SMTP id x11mr3987037otb.184.1559253290577; Thu, 30 May 2019 14:54:50 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id u8sm1544640otk.53.2019.05.30.14.54.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:54:50 -0700 (PDT) Subject: [RFC PATCH 10/11] virtio-balloon: Add support for aerating memory via bubble hinting From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:54:48 -0700 Message-ID: <20190530215448.13974.59362.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck Add support for aerating memory using the bubble hinting feature provided by virtio-balloon. Bubble hinting differs from the regular balloon functionality in that is is much less durable than a standard memory balloon. Instead of creating a list of pages that cannot be accessed the pages are only inaccessible while they are being indicated to the virtio interface. Once the interface has acknowledged them they are placed back into their respective free lists and are once again accessible by the guest system. Signed-off-by: Alexander Duyck --- drivers/virtio/Kconfig | 1 drivers/virtio/virtio_balloon.c | 89 +++++++++++++++++++++++++++++++++++ include/uapi/linux/virtio_balloon.h | 1 3 files changed, 90 insertions(+), 1 deletion(-) diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig index 023fc3bc01c6..9cdaccf92c3a 100644 --- a/drivers/virtio/Kconfig +++ b/drivers/virtio/Kconfig @@ -47,6 +47,7 @@ config VIRTIO_BALLOON tristate "Virtio balloon driver" depends on VIRTIO select MEMORY_BALLOON + select AERATION ---help--- This driver supports increasing and decreasing the amount of memory within a KVM guest. diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index 44339fc87cc7..e1399991bc1f 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -18,6 +18,7 @@ #include #include #include +#include /* * Balloon device works in 4K page units. So each page is pointed to by @@ -45,6 +46,7 @@ enum virtio_balloon_vq { VIRTIO_BALLOON_VQ_DEFLATE, VIRTIO_BALLOON_VQ_STATS, VIRTIO_BALLOON_VQ_FREE_PAGE, + VIRTIO_BALLOON_VQ_HINTING, VIRTIO_BALLOON_VQ_MAX }; @@ -52,9 +54,16 @@ enum virtio_balloon_config_read { VIRTIO_BALLOON_CONFIG_READ_CMD_ID = 0, }; +#define VIRTIO_BUBBLE_ARRAY_HINTS_MAX 32 +struct virtio_bubble_page_hint { + __virtio32 pfn; + __virtio32 size; +}; + struct virtio_balloon { struct virtio_device *vdev; - struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq; + struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq, + *hinting_vq; /* Balloon's own wq for cpu-intensive work items */ struct workqueue_struct *balloon_wq; @@ -107,6 +116,11 @@ struct virtio_balloon { unsigned int num_pfns; __virtio32 pfns[VIRTIO_BALLOON_ARRAY_PFNS_MAX]; + /* The array of PFNs we are hinting on */ + unsigned int num_hints; + struct virtio_bubble_page_hint hints[VIRTIO_BUBBLE_ARRAY_HINTS_MAX]; + struct aerator_dev_info a_dev_info; + /* Memory statistics */ struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR]; @@ -151,6 +165,54 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq) } +void virtballoon_aerator_react(struct aerator_dev_info *a_dev_info) +{ + struct virtio_balloon *vb = container_of(a_dev_info, + struct virtio_balloon, + a_dev_info); + struct virtqueue *vq = vb->hinting_vq; + struct scatterlist sg; + unsigned int unused; + struct page *page; + + vb->num_hints = 0; + + list_for_each_entry(page, &a_dev_info->batch_reactor, lru) { + struct virtio_bubble_page_hint *hint; + unsigned int size; + + hint = &vb->hints[vb->num_hints++]; + hint->pfn = cpu_to_virtio32(vb->vdev, + page_to_balloon_pfn(page)); + size = VIRTIO_BALLOON_PAGES_PER_PAGE << page_private(page); + hint->size = cpu_to_virtio32(vb->vdev, size); + } + + /* We shouldn't have been called if there is nothing to process */ + if (WARN_ON(vb->num_hints == 0)) + return; + + /* Detach all the used buffers from the vq */ + while (virtqueue_get_buf(vq, &unused)) + ; + + sg_init_one(&sg, vb->hints, + sizeof(vb->hints[0]) * vb->num_hints); + + /* + * We should always be able to add one buffer to an + * empty queue. + */ + virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL); + virtqueue_kick(vq); +} + +static void aerator_settled(struct virtqueue *vq) +{ + /* Drain the current aerator contents, refill, and start next cycle */ + aerator_cycle(); +} + static void set_page_pfns(struct virtio_balloon *vb, __virtio32 pfns[], struct page *page) { @@ -475,6 +537,7 @@ static int init_vqs(struct virtio_balloon *vb) names[VIRTIO_BALLOON_VQ_DEFLATE] = "deflate"; names[VIRTIO_BALLOON_VQ_STATS] = NULL; names[VIRTIO_BALLOON_VQ_FREE_PAGE] = NULL; + names[VIRTIO_BALLOON_VQ_HINTING] = NULL; if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) { names[VIRTIO_BALLOON_VQ_STATS] = "stats"; @@ -486,11 +549,19 @@ static int init_vqs(struct virtio_balloon *vb) callbacks[VIRTIO_BALLOON_VQ_FREE_PAGE] = NULL; } + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_HINTING)) { + names[VIRTIO_BALLOON_VQ_HINTING] = "hinting_vq"; + callbacks[VIRTIO_BALLOON_VQ_HINTING] = aerator_settled; + } + err = vb->vdev->config->find_vqs(vb->vdev, VIRTIO_BALLOON_VQ_MAX, vqs, callbacks, names, NULL, NULL); if (err) return err; + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_HINTING)) + vb->hinting_vq = vqs[VIRTIO_BALLOON_VQ_HINTING]; + vb->inflate_vq = vqs[VIRTIO_BALLOON_VQ_INFLATE]; vb->deflate_vq = vqs[VIRTIO_BALLOON_VQ_DEFLATE]; if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_STATS_VQ)) { @@ -929,12 +1000,25 @@ static int virtballoon_probe(struct virtio_device *vdev) if (err) goto out_del_balloon_wq; } + + vb->a_dev_info.react = virtballoon_aerator_react; + vb->a_dev_info.capacity = VIRTIO_BUBBLE_ARRAY_HINTS_MAX; + INIT_LIST_HEAD(&vb->a_dev_info.batch_reactor); + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_HINTING)) { + err = aerator_startup(&vb->a_dev_info); + if (err) + goto out_unregister_shrinker; + } + virtio_device_ready(vdev); if (towards_target(vb)) virtballoon_changed(vdev); return 0; +out_unregister_shrinker: + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) + virtio_balloon_unregister_shrinker(vb); out_del_balloon_wq: if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) destroy_workqueue(vb->balloon_wq); @@ -963,6 +1047,8 @@ static void virtballoon_remove(struct virtio_device *vdev) { struct virtio_balloon *vb = vdev->priv; + if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_HINTING)) + aerator_shutdown(); if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) virtio_balloon_unregister_shrinker(vb); spin_lock_irq(&vb->stop_update_lock); @@ -1032,6 +1118,7 @@ static int virtballoon_validate(struct virtio_device *vdev) VIRTIO_BALLOON_F_DEFLATE_ON_OOM, VIRTIO_BALLOON_F_FREE_PAGE_HINT, VIRTIO_BALLOON_F_PAGE_POISON, + VIRTIO_BALLOON_F_HINTING, }; static struct virtio_driver virtio_balloon_driver = { diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h index a1966cd7b677..2b0f62814e22 100644 --- a/include/uapi/linux/virtio_balloon.h +++ b/include/uapi/linux/virtio_balloon.h @@ -36,6 +36,7 @@ #define VIRTIO_BALLOON_F_DEFLATE_ON_OOM 2 /* Deflate balloon on OOM */ #define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3 /* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4 /* Guest is using page poisoning */ +#define VIRTIO_BALLOON_F_HINTING 5 /* Page hinting virtqueue */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12 From patchwork Thu May 30 21:54:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10969303 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 93BFB912 for ; Thu, 30 May 2019 21:55:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8577F28A60 for ; Thu, 30 May 2019 21:55:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 79F4A28BD3; Thu, 30 May 2019 21:55:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F342428C19 for ; Thu, 30 May 2019 21:55:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 777F46B027D; Thu, 30 May 2019 17:54:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7280F6B027E; Thu, 30 May 2019 17:54:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C89A6B027F; Thu, 30 May 2019 17:54:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) by kanga.kvack.org (Postfix) with ESMTP id 307AC6B027D for ; Thu, 30 May 2019 17:54:59 -0400 (EDT) Received: by mail-ot1-f72.google.com with SMTP id b64so380504otc.3 for ; Thu, 30 May 2019 14:54:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=tFR5eWyyHKLzv/rG+/k1EP28VfnvmUHd1en3f8cuet4=; b=jpE5UKzeX5B4IRI3xybSLhugyRd+2eriN63XPhH+uyZvXANlFGllaZijJ1BQpTAmNe elpjgWZG+J8huUYRk77YbcZNG1iZBSE8HT+NbVHZy8aIzAYFMbtepDY9mGGXrCb2T4n6 0akdwpIgZCPN6pcog3v9y2RAObn47DAZBPc7Bl2VLU4qrVShGa2CErOATxXNiQ2yZGyP 3gnuMaCHWANnCsK5vvV0Ugt1k2CWPpZADfhckZRD11YKvKoPNKWqj4LTVYqjEb1k6k4L Zh+OVke1N7zEqjUr1fAcuTi3ZV9RDS5rllejZE6nnUR9b5QxMgUPuDYQ8YdUlb3VwnLg IjJw== X-Gm-Message-State: APjAAAVKEDaoxblemQFkdtCme9gGk7cJJzporeIi1EWAARYFc3xQtBUX NzVF33thNxK9xGKUdSF52z3ckSaXlEJy4IL2NgAqbFLN+aJnjbs0lUpUSVdyd/nj8afQP3oQJ0R OCutgGna2hmsj4D8r8s3TF3d++fRgm1TJOoI8ox6E+MwCFu81OXpMf9YdGBACUCmQmg== X-Received: by 2002:a9d:411e:: with SMTP id o30mr4289711ote.122.1559253298915; Thu, 30 May 2019 14:54:58 -0700 (PDT) X-Received: by 2002:a9d:411e:: with SMTP id o30mr4289683ote.122.1559253298381; Thu, 30 May 2019 14:54:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559253298; cv=none; d=google.com; s=arc-20160816; b=mwj9Gw+eYREJWl/qMQ19bX5TGLroUky+akMZC9ewVkS9Y1HDAmtU4eEZKWAGjfOrKz 8vejTQgpYvsCEl91cMKULL4OkPslfzLOO8TzWrnTcPcBpS5P6ywVZshr4xrMLk3z1Blk U2pdtLHf78AGlNqmOQbIUa4toq8kL1WrX5lh2B4R4X5vKDtdlZNO0rxR0WYC4/E562fQ kZINjArrp9r8eh3+amY3gSM/PR1d9bVTR80jdeVOzslIt8YLLAcIhrVgsTFTge9Mf+qU WBnGuI0fh5Ramo7KUBZCc0osMoHHeejbuz3hhHIflS9W/qJgJJRGypzbY6jMl6cRudL5 /uHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=tFR5eWyyHKLzv/rG+/k1EP28VfnvmUHd1en3f8cuet4=; b=BCHTL1etF53GcHaGiza9ZtY2TJ+GAEu8CvQSg+wDcQhtKbK1BIGTH4MkOVo7PdVQ1I rJKaQZxD4wtzhJ59FibzQ/GWgdBDTHot2ku1Dy7pNlWZzC3J5NExBgShD1/NNB+y/pEL JARe/R1gBRyW9Ntc423SgKyPFi3SbYmoH16iRezxThHBfBobj8iZyvZjFEAFSPiUboji xKrP2l/eo134KR6SQ4VshL6yoCfuB6H/dqdu0NBJC3y7Uf5wn60CdLGeTf7bONC/1hQR nxqmwg2tpPu65nNBbGLmI3ROwqky/uMQqno9ysswkgZvpLKxfZLj3xHmCL98HJFjYWOF 0X8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=QntIfn3Q; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id o188sor1233818oih.142.2019.05.30.14.54.58 for (Google Transport Security); Thu, 30 May 2019 14:54:58 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=QntIfn3Q; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=tFR5eWyyHKLzv/rG+/k1EP28VfnvmUHd1en3f8cuet4=; b=QntIfn3QP7w8OAf3KeLsA35GzvqasynhBr/LPKQ9UMSeS7sqkj18UbNYx/lJnWUJpx f9XhduhHSqYQrgukNTzCY6WtbcIgGHsNYgnGj2IhxiM0AVHY2+kyTEyxdSCPnZMVDLIx 9F25FqTBOpb+X9YuJjyHH3ZeN/n7u048oiFLngFfpUu8oI0RouCxcc88D1IA4PwTWbV6 YeeIFLwG3GLDmk1UTKsyorsyB8oh3Cc+b4Ql1DRnh02Uyw3Mddx82RmNGlqoHGX8YCHx neY4bedgWaliCWNuwj2x5E4P1f3BsPX6mffX4wlZD7UVun/2Shv7YNn8yRoAMtSRlhmk Enrw== X-Google-Smtp-Source: APXvYqxfGBCA97YkaWM4wjgbjBJNwlQrAjyiLnkd5V2gUCN0ncNK3/MIrN/Ma+crH/dforiEy7/ROg== X-Received: by 2002:aca:4e42:: with SMTP id c63mr4187588oib.170.1559253298022; Thu, 30 May 2019 14:54:58 -0700 (PDT) Received: from localhost.localdomain (50-126-100-225.drr01.csby.or.frontiernet.net. [50.126.100.225]) by smtp.gmail.com with ESMTPSA id 33sm1412918otb.56.2019.05.30.14.54.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 May 2019 14:54:57 -0700 (PDT) Subject: [RFC PATCH 11/11] mm: Add free page notification hook From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Thu, 30 May 2019 14:54:55 -0700 Message-ID: <20190530215455.13974.87717.stgit@localhost.localdomain> In-Reply-To: <20190530215223.13974.22445.stgit@localhost.localdomain> References: <20190530215223.13974.22445.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck Add a hook so that we are notified when a new page is available. We will use this hook to notify the virtio aeration system when we have achieved enough free higher-order pages to justify the process of pulling some pages and hinting on them. Signed-off-by: Alexander Duyck --- arch/x86/include/asm/page.h | 11 +++++++++++ include/linux/gfp.h | 4 ++++ mm/page_alloc.c | 2 ++ 3 files changed, 17 insertions(+) diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index 7555b48803a8..dfd546230120 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -18,6 +18,17 @@ struct page; +#ifdef CONFIG_AERATION +#include + +#define HAVE_ARCH_FREE_PAGE_NOTIFY +static inline void +arch_free_page_notify(struct page *page, struct zone *zone, int order) +{ + aerator_notify_free(page, zone, order); +} + +#endif #include extern struct range pfn_mapped[]; extern int nr_pfn_mapped; diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 407a089d861f..d975e7eabbf8 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -459,6 +459,10 @@ static inline struct zonelist *node_zonelist(int nid, gfp_t flags) #ifndef HAVE_ARCH_FREE_PAGE static inline void arch_free_page(struct page *page, int order) { } #endif +#ifndef HAVE_ARCH_FREE_PAGE_NOTIFY +static inline void +arch_free_page_notify(struct page *page, struct zone *zone, int order) { } +#endif #ifndef HAVE_ARCH_ALLOC_PAGE static inline void arch_alloc_page(struct page *page, int order) { } #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e3800221414b..104763034ce3 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -999,6 +999,8 @@ static inline void __free_one_page(struct page *page, add_to_free_area_tail(page, area, migratetype); else add_to_free_area(page, area, migratetype); + + arch_free_page_notify(page, zone, order); } /*