From patchwork Wed Oct 18 17:11:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Ryabinin X-Patchwork-Id: 10015105 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3D73D601E7 for ; Wed, 18 Oct 2017 17:09:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24F3229246 for ; Wed, 18 Oct 2017 17:09:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1A70C2925D; Wed, 18 Oct 2017 17:09:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9B84029246 for ; Wed, 18 Oct 2017 17:09:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZZ4w5kBujWwvBpZavWAH2MFyqFH7F3Cv4P44UZoAYHg=; b=HYFX+21KJPfWojfUQxoJDNcUaj K9+E7Xhq0lABdXw7JOoaEkJh68MEL8IU2vvRQ/u8hqVBPlmaoGsXQ302yuNPXUlpHqLUm/sYGMI+5 8ebTTrJDYXsqxC8pyRwtyJL2Ogpko3549KayqJ7ZlSWiX7WthTMEOK+B/xx/0VvBC94Si4JEvQnO8 FfQit41inG4zAnn4OM+8sB0RikggD3G00E0pnZnnglVaStIks3Pk6pqUq4BtS5fx0jsf0qqcMVEN6 V0yM/mqrtHs6Fan/Zq12SnyotFEvEVuu7oxpmx7f0XjqdSkNR5MHagXtKS8mo/zCu3QvQ4joSVPts xIAsWDWg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e4rqE-0000ba-5l; Wed, 18 Oct 2017 17:09:14 +0000 Received: from mail-ve1eur01on0119.outbound.protection.outlook.com ([104.47.1.119] helo=EUR01-VE1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e4rpU-0008Ig-Vp for linux-arm-kernel@lists.infradead.org; Wed, 18 Oct 2017 17:09:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=YO9r0LaI2J+nQKq5oBsGriE688KlmXeCw+ssEh0NL8w=; b=dAtGvlOTC/JVt7uUxhtDxrBQ6+6Psra1lT53EzuQALj3FwidM0Cynqa7tbRMHkWee+lSDFlT0rlChPp0Mo7tiRLxN2DheWReHM8sM2oW5T/RKwECLQ1HJAAfD0QAcEBqHWR/zG83j7owPtfo7aY+GKVrXqhKQgPdKdH1TmrxXhQ= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=aryabinin@virtuozzo.com; Received: from [172.16.25.12] (195.214.232.6) by DB6PR08MB2823.eurprd08.prod.outlook.com (2603:10a6:6:1d::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Wed, 18 Oct 2017 17:08:04 +0000 Subject: Re: [PATCH v12 07/11] x86/kasan: add and use kasan_map_populate() To: Pavel Tatashin , linux-kernel@vger.kernel.org, sparclinux@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, kasan-dev@googlegroups.com, borntraeger@de.ibm.com, heiko.carstens@de.ibm.com, davem@davemloft.net, willy@infradead.org, mhocko@kernel.org, ard.biesheuvel@linaro.org, mark.rutland@arm.com, will.deacon@arm.com, catalin.marinas@arm.com, sam@ravnborg.org, mgorman@techsingularity.net, akpm@linux-foundation.org, steven.sistare@oracle.com, daniel.m.jordan@oracle.com, bob.picco@oracle.com References: <20171013173214.27300-1-pasha.tatashin@oracle.com> <20171013173214.27300-8-pasha.tatashin@oracle.com> From: Andrey Ryabinin Message-ID: Date: Wed, 18 Oct 2017 20:11:09 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20171013173214.27300-8-pasha.tatashin@oracle.com> Content-Language: en-US X-Originating-IP: [195.214.232.6] X-ClientProxiedBy: AM5P189CA0014.EURP189.PROD.OUTLOOK.COM (2603:10a6:206:15::27) To DB6PR08MB2823.eurprd08.prod.outlook.com (2603:10a6:6:1d::26) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: afeb617e-b931-45db-d06a-08d5164acc8d X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:DB6PR08MB2823; X-Microsoft-Exchange-Diagnostics: 1; DB6PR08MB2823; 3:cdFTlY2+6z3qiINzzREtxwE7F8ogtJW/1xV0Ph3xNd3AUnEOR3kSZDMIEpWGvyysx+qy/Xsz8v0CTIyfUKrgUr3+yXnbwVVklJlvIZ2VGgFu4oKDuSQarYtVoeMjPJ/vhXkYeXPXv6jOc7bdzPXkh/2/IXTQVwXwDRU6ZBprAEFXRLfxHL21BM2y6rdVLq/FM7wDFpsUBlDNY4akvBFsNC0x8xtnIAbs0JKYX1MpcKbHW8l/aZfloqICVBQjZ19b; 25:YrkKOFwbaCkk0J0gIIq9Abe91HUfR/WlB0YHuyDLM1Z+ICGa/oDO5/5sZhZ1T/WO9PXI5pa6yx8PiZAty84lGBCEcEwJqkrb8MoqhzLXjU/HRZIcy1dYa5JMEbqZ997/h3c4rHDc8EPkC0VPmc7zwiqQkZa23GuNWnsC193ZiKCQBSI74jduYHEujS2tdoGIDiBPoJaZPtoG9vwmo/TwAWBsqd4kcp9NWldgYDBBOQtAh5KdRyCTSP15NtvVAAA86+EvJV59/Ufq65tPG8fD/cegPn6jENz7WRopkNpcxd3uGvmjmvuDwvfU8bu6nDaMT7GGB3/TqYtnIWOGlTd5mw==; 31:Cwy0H9LBWdiw0N1PQWO7lQPBwQFYZU8I6RG4zfu1OxNkuAbv+pa/XC2+jcKBL7QZxm1iiIBP0Enq8aSy3rGfoTLiBQ2YatO7QXBEjirKIyn2trNm5k5hQAK4Bcaxjba3uGmIKknDTYPfc65H98KRF8o4RMWuROpOHHJNAcW92UyQQ/Nd8b/CNpuTPQLnr/ysi82hT3k++a/+cUPpYmzwoXxbfB/9VLZ1POPWubG6ZJc= X-MS-TrafficTypeDiagnostic: DB6PR08MB2823: X-Microsoft-Exchange-Diagnostics: 1; DB6PR08MB2823; 20:hXPMVYLCuB45ke3gSpjjSqbw0U1OqMS//mjpy3/FY+4Fej2Ojx73et8/5dcIpLZ3vPE6jZECdQd4dyyPTmdoMW7WGw4Cubr/ArGQsiM26EF40o9ax/YTYSoD7aRjrxv5jfGAuyEbXhHnYEX2nsTN8NYvPI1reUiiuRkgTR72cgIVzAg9iP2Q7EmTSCOdIm6iUo5mGlaIExLCXtDL28mkdKMozh/G8bs1eukmcbmnOrKvT4eIWgT+coaKQw9LpAFzWwBpWhT4M7bWb7bbk34szXZ8qN1bVk7sqZeTziSvukusZRSAkOSA52nU9oVsFLusyhSfwDIJk5cxs04ZPk7vhN1/MPjaVAVGMtSPuTsxT8MoThKi9HTGqxKse+vq25WEJLRYyIljXGxkyi43CDx5j4v4C/RhJw2m5iIhXLUNFQY=; 4:j54vdvzLM2cztvmIJS3WnPbtq8CjvnhU7tMTe4EmteIjH+q9/Fa+b6E31khR77XT4GII0927lrc+MzyxSN2aUB9FOnGO0l1CgrFpiAWTqSqD+N2hnCddjJeQoYa0ocoJkAnk35XzCxilU4ugUKPM02PvI8qIZvcQl8VQafStnrNbuWDlHz6ZElAVmktMtGYp0KudBaUB8QOIiRlXJRsNVY2pywkB0T/oZjWoNruDleruZfu8iYS2JhieaapJw5hD X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3002001)(100000703101)(100105400095)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123555025)(20161123560025)(20161123564025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DB6PR08MB2823; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DB6PR08MB2823; X-Forefront-PRVS: 0464DBBBC4 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(6049001)(6009001)(346002)(376002)(24454002)(377454003)(199003)(189002)(6116002)(101416001)(50986999)(58126008)(230700001)(76176999)(2906002)(68736007)(65806001)(189998001)(16526018)(47776003)(5660300001)(2950100002)(66066001)(3846002)(81156014)(64126003)(83506001)(316002)(54356999)(16576012)(8676002)(81166006)(305945005)(8936002)(65826007)(6246003)(25786009)(65956001)(106356001)(478600001)(33646002)(7736002)(105586002)(36756003)(23676002)(6486002)(50466002)(53936002)(77096006)(7416002)(97736004)(53546010)(31696002)(86362001)(229853002)(31686004)(921003)(1121003); DIR:OUT; SFP:1102; SCL:1; SRVR:DB6PR08MB2823; H:[172.16.25.12]; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: virtuozzo.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtEQjZQUjA4TUIyODIzOzIzOnZJTTJyaUF3Q1R2ZHhzc0V1T2o5MUMzZDE5?= =?utf-8?B?YUc5MW9zMENpRldHS2ljdGY5MXlnUmxPM0dPMTZJSDV2TjZ3VDNPWjVrcWVT?= =?utf-8?B?ZmtSckszSitNV0JSNk42NHNaS2ZSaHJvOG5XRG5VZ1JQU3FTVW5lT2VqaGoy?= =?utf-8?B?VWJjL0JXY3B0Z2FLUXNyTVFnNkllTmRCbjJlZG1rZmlETWpDRjQ5YWtxbWF0?= =?utf-8?B?MGx4U2lld0p4Q3RlYnBZaU83SEVBRmN2OW95MDVaTkN5Wldua0VzdlpUTDJO?= =?utf-8?B?aXFqd0JkMXVyekIvYlpTTXlxVFdFdU5sbVpnYjNGRW5JcXdlb0FsQ01TNnVG?= =?utf-8?B?MnVTc0htTHRXWjVPRVJ2Vjh2NFhzZ2h3NXBpakpHOVNaeXM1Y09kMGFQbVYr?= =?utf-8?B?M20vTXdWR0EyV1JWRVUwSm1IZUJrUDlKcHM0NkN4Mis2bUtJTE40d3F3V2g2?= =?utf-8?B?L0tYTTdFek9YeGowN1JOWko2TjZ3cmYrQitkTTY5RlJ1bmRQQ1NzTlZ1T1ox?= =?utf-8?B?SkRyZlRwNDhMMkJrWHJWQXhpVElhbWJzWi9GZGE2YjBNSVhYSXJGcTZBVTVa?= =?utf-8?B?dWdIMFZ2Vy9NbWF2UGx6KzFzOGFUdk5ncHdXZzB3bldad1VBSGIrc3hoczdM?= =?utf-8?B?Y094NklDSW81UzRuNzdBS2ZSZ0hRM3hzUlUwbTE1aDZnR2MrTW5zMmo4bWxQ?= =?utf-8?B?Zi9oR0dablh3Vml0REhLV0xIdHBXVnBhMVZZb3JvbUU2MG9IQW91V2VudDk2?= =?utf-8?B?d3B4QWFHNXliakxYdjBoK2VuQXE0TFo3VVhMRFFPS1d1d3VDcTdHT2Z0MU9O?= =?utf-8?B?U3hIemV4bGZyYlcyM2FGSEl3cFRoTWNVYXlhd05yVzFrYzhhNFMrQTdSMUNS?= =?utf-8?B?Z2xhd2FJeS93Ym1vNFdOb053VFgwVElLSVI3VFlWdGlkUzNyMHFGbTlNOWZW?= =?utf-8?B?a2NKVWZjSlRsamZ6dGxzM093bVdlQ1hxYTdHK3RWUWV4OWV2ZkQ5ZWhyRVkw?= =?utf-8?B?NStxTHVScmdDc3ZCdTNjNWtHanFYMnpiU2ZRelRiem41aWVkMUE0aW9iUEto?= =?utf-8?B?OTFCSmx6aTNwbG9TWUlrRjQzd0ZndGFCM09UZmplQU5XZmV6V0VEaTlIZGtm?= =?utf-8?B?R2JvYU81b2lKbjdtbG9nZnl5Yy8wRGtVNHJIUzRFMHJ5eHNTWkJYYjJLWmFQ?= =?utf-8?B?Q3BUM3ZKa1Bwa2QyVGE3azJObWxOeTQwcFY0VmtmMjhBSWxTS2VBNURObGtq?= =?utf-8?B?TXhEdDNjQm5XUEZQdzFybnl4Z3pLV0RmWWpzVXBlRE1pTm9ncVpzby95NmF6?= =?utf-8?B?dkU3U25aUTZpSU9ROC9qc3dqdERzendIeUN5cUYzWHFyZzEwRk1EUG0xUUlX?= =?utf-8?B?eFkwUDdaaUxDaStTWHY3R1VOU0EyR3pIQzR5cEFwbkhnRmFPQ0tKdFdQaHI2?= =?utf-8?B?eWQ4SUJsRzRaZUtUV2F1cmdLYUZGK2luZnN5Y254eWlQb0RUQkR0RlUxTHFB?= =?utf-8?B?aGhtSkl1Z0gvWkxrUGZNbldOTEJVc1NrNEwwUTBBMGhYTWRGYWlBNnJERHgr?= =?utf-8?B?dlkvNlBSaHRJM2xuZ1luVVN2V3NrZzU5VGd1QzZrcVo2REVHdjdvdkorZmZ6?= =?utf-8?B?NDJpTktwU2hXRGJJZk5iSWRrTGlra3RYd1VKSUNVTmFwejNyUkpaU1dvSERZ?= =?utf-8?B?ajZKVDVucGc4aitwbUxuNjlzUUhBZ0l4S243VUR1K1padUtxZFhkWUpxZWRJ?= =?utf-8?B?VWF6dkptYXFFcXd5Nm9FbFl5ZitlRlRLR2ZXMXJnejh4NHgwZEY5TEhvVFVU?= =?utf-8?Q?g5ClApzAzIAKc?= X-Microsoft-Exchange-Diagnostics: 1; DB6PR08MB2823; 6:otOqAkzxEnq1sp3HrhXt7PNgI6VuzuDeWkiU1ngyz44XVqIVEwt1f7ys71yFscESETbQYQvABvj6Vv6ZDSSDxf+QfYPE9pTxernWmcxDms7XGV6OVXHIX9o3zAL3i+cJyHb9JIbDSi4jTutzx/ct93iUvw4ZbrHwAcU6CF+JU8F40R5WMvwsCCboGvaKbMczkHvZefKphRD57TlyrLt1a86NRoM6edkBmqHeSdlxKAzmRcaylpz6pxryHa0aAFYQxxWAW4X1b3OnKNJu2Ke0Yuxe4snRNI3fU/X8LeCMbMbGbJ6jAFA/b3lNOuQHvGJdbpVz0VrpXznMQnRUdCUL2A==; 5:8RcTSv0HuclX2DeE0XdlCrP2klGBQqDT1P4Fo7zvyFZvYhVQ2pt6K2A4JhXkAiHTK+voW3h6EFyKYVTnuVAvPU3+7GYgRwMSfFZnby4inrTW8YNNEKoiJwHM3LBSz/sBT5ZSMJZqOl1Z9g+6dbjCr+V9nsTbaLj7hTby7cMkPkk=; 24:31OvMCyXv2QAYOd0Sxb9jxwW4ozeH118hfj7++SLbRT4UbXLbIwzGjyilLgEOieG04lqFNEQCX5VLkLit464azITwVZ+46OqdtwesoktZ3Y=; 7:EtsGoyPwohqvYsD549HWTrUb/Umiw83hmn2bRWx+o9JNkBw/qEugrCaOABcxtHH5pQqwEJOgRlsmVOfe3KwBByAvRipT0tq7/prKEWYyw/nHcu1sJERxbWMhSbwOGTLJuAF3szEXcqPlNZANQDDYLKQ2z4iz5JtDjJ2kcjCNoDl+KWDL7dLmf3Ii2xx+TB66Fdj2VGQ5ZrdmCTcfPBxdhISKvY/iR6Rb8o4eldmT1z8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DB6PR08MB2823; 20:QQJrUnSP0SDiTqeJb/81vBm4RM++uIN3X32b+i9XIQ8MRKAyw4hUJwcs2kVGw47yIGdKvcBcnRW5uAjQhEVc3gu+YwZD74Ry+A3IyXMdzU7I374Q9Owlr95+xNbCg1f5KyK9LPuD6yytG9D94ggxGD8Wm85Szy3Rwr+aBSjXNC0= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2017 17:08:04.0842 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR08MB2823 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171018_100830_044255_25C03F24 X-CRM114-Status: GOOD ( 17.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On 10/13/2017 08:32 PM, Pavel Tatashin wrote: > During early boot, kasan uses vmemmap_populate() to establish its shadow > memory. But, that interface is intended for struct pages use. > > Because of the current project, vmemmap won't be zeroed during allocation, > but kasan expects that memory to be zeroed. We are adding a new > kasan_map_populate() function to resolve this difference. > > Therefore, we must use a new interface to allocate and map kasan shadow > memory, that also zeroes memory for us. > Nah, we should do the same as with arm64 here. The patch bellow works for me. Could you please make sure that it works for you as well? Just in case. From: Andrey Ryabinin Subject: x86/mm/kasan: don't use vmemmap_populate() to initialize shadow The kasan shadow is currently mapped using vmemmap_populate() since that provides a semi-convenient way to map pages into init_top_pgt. However, since that no longer zeroes the mapped pages, it is not suitable for kasan, which requires zeroed shadow memory. Add kasan_populate_shadow() interface and use it instead of vmemmap_populate(). Besides, this allows us to take advantage of gigantic pages and use them to populate the shadow, which should save us some memory wasted on page tables and reduce TLB pressure. Signed-off-by: Andrey Ryabinin --- arch/x86/Kconfig | 2 +- arch/x86/mm/kasan_init_64.c | 141 ++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 136 insertions(+), 7 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 8153cf40e5fe..e3847e472bd7 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -108,7 +108,7 @@ config X86 select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE select HAVE_ARCH_JUMP_LABEL - select HAVE_ARCH_KASAN if X86_64 && SPARSEMEM_VMEMMAP + select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index bc84b73684b7..f8ea02571494 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -3,19 +3,148 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include #include extern struct range pfn_mapped[E820_MAX_ENTRIES]; -static int __init map_range(struct range *range) +static __init void *early_alloc(size_t size, int nid) +{ + return memblock_virt_alloc_try_nid_nopanic(size, size, + __pa(MAX_DMA_ADDRESS), BOOTMEM_ALLOC_ACCESSIBLE, nid); +} + +static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr, + unsigned long end, int nid) +{ + pte_t *pte; + + if (pmd_none(*pmd)) { + void *p; + + if (boot_cpu_has(X86_FEATURE_PSE) && + ((end - addr) == PMD_SIZE) && + IS_ALIGNED(addr, PMD_SIZE)) { + p = early_alloc(PMD_SIZE, nid); + if (p && pmd_set_huge(pmd, __pa(p), PAGE_KERNEL)) + return; + else if (p) + memblock_free(__pa(p), PMD_SIZE); + } + + p = early_alloc(PAGE_SIZE, nid); + pmd_populate_kernel(&init_mm, pmd, p); + } + + pte = pte_offset_kernel(pmd, addr); + do { + pte_t entry; + void *p; + + if (!pte_none(*pte)) + continue; + + p = early_alloc(PAGE_SIZE, nid); + entry = pfn_pte(PFN_DOWN(__pa(p)), PAGE_KERNEL); + set_pte_at(&init_mm, addr, pte, entry); + } while (pte++, addr += PAGE_SIZE, addr != end); +} + + +static void __init kasan_populate_pud(pud_t *pud, unsigned long addr, + unsigned long end, int nid) +{ + pmd_t *pmd; + unsigned long next; + + if (pud_none(*pud)) { + void *p; + + if (boot_cpu_has(X86_FEATURE_GBPAGES) && + ((end - addr) == PUD_SIZE) && + IS_ALIGNED(addr, PUD_SIZE)) { + p = early_alloc(PUD_SIZE, nid); + if (p && pud_set_huge(pud, __pa(p), PAGE_KERNEL)) + return; + else if (p) + memblock_free(__pa(p), PUD_SIZE); + } + + p = early_alloc(PAGE_SIZE, nid); + pud_populate(&init_mm, pud, p); + } + + pmd = pmd_offset(pud, addr); + do { + next = pmd_addr_end(addr, end); + if (!pmd_large(*pmd)) + kasan_populate_pmd(pmd, addr, next, nid); + } while (pmd++, addr = next, addr != end); +} + +static void __init kasan_populate_p4d(p4d_t *p4d, unsigned long addr, + unsigned long end, int nid) +{ + pud_t *pud; + unsigned long next; + + if (p4d_none(*p4d)) { + void *p = early_alloc(PAGE_SIZE, nid); + p4d_populate(&init_mm, p4d, p); + } + + pud = pud_offset(p4d, addr); + do { + next = pud_addr_end(addr, end); + if (!pud_large(*pud)) + kasan_populate_pud(pud, addr, next, nid); + } while (pud++, addr = next, addr != end); +} + +static void __init kasan_populate_pgd(pgd_t *pgd, unsigned long addr, + unsigned long end, int nid) +{ + void *p; + p4d_t *p4d; + unsigned long next; + + if (pgd_none(*pgd)) { + p = early_alloc(PAGE_SIZE, nid); + pgd_populate(&init_mm, pgd, p); + } + + p4d = p4d_offset(pgd, addr); + do { + next = p4d_addr_end(addr, end); + kasan_populate_p4d(p4d, addr, next, nid); + } while (p4d++, addr = next, addr != end); +} + +static void __init kasan_populate_shadow(unsigned long addr, unsigned long end, + int nid) +{ + pgd_t *pgd; + unsigned long next; + + addr = addr & PAGE_MASK; + end = round_up(end, PAGE_SIZE); + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + kasan_populate_pgd(pgd, addr, next, nid); + } while (pgd++, addr = next, addr != end); +} + +static void __init map_range(struct range *range) { unsigned long start; unsigned long end; @@ -23,7 +152,7 @@ static int __init map_range(struct range *range) start = (unsigned long)kasan_mem_to_shadow(pfn_to_kaddr(range->start)); end = (unsigned long)kasan_mem_to_shadow(pfn_to_kaddr(range->end)); - return vmemmap_populate(start, end, NUMA_NO_NODE); + kasan_populate_shadow(start, end, early_pfn_to_nid(range->start)); } static void __init clear_pgds(unsigned long start, @@ -129,16 +258,16 @@ void __init kasan_init(void) if (pfn_mapped[i].end == 0) break; - if (map_range(&pfn_mapped[i])) - panic("kasan: unable to allocate shadow!"); + map_range(&pfn_mapped[i]); } + kasan_populate_zero_shadow( kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM), kasan_mem_to_shadow((void *)__START_KERNEL_map)); - vmemmap_populate((unsigned long)kasan_mem_to_shadow(_stext), + kasan_populate_shadow((unsigned long)kasan_mem_to_shadow(_stext), (unsigned long)kasan_mem_to_shadow(_end), - NUMA_NO_NODE); + early_pfn_to_nid(__pa(_stext))); kasan_populate_zero_shadow(kasan_mem_to_shadow((void *)MODULES_END), (void *)KASAN_SHADOW_END);