From patchwork Fri Sep 6 14:53:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11135459 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6DE611395 for ; Fri, 6 Sep 2019 14:53:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 39EF6206A5 for ; Fri, 6 Sep 2019 14:53:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Mag0NQ/B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 39EF6206A5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3F17F6B0008; Fri, 6 Sep 2019 10:53:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3C8156B000A; Fri, 6 Sep 2019 10:53:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DE066B000C; Fri, 6 Sep 2019 10:53:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id 0BD3F6B0008 for ; Fri, 6 Sep 2019 10:53:31 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id ADD2B181AC9AE for ; Fri, 6 Sep 2019 14:53:30 +0000 (UTC) X-FDA: 75904789380.19.corn58_58ccd3e67e337 X-Spam-Summary: 2,0,0,28f5037ca4b378d9,d41d8cd98f00b204,alexander.duyck@gmail.com,:nitesh@redhat.com:kvm@vger.kernel.org:mst@redhat.com:david@redhat.com:dave.hansen@intel.com:linux-kernel@vger.kernel.org:willy@infradead.org:mhocko@kernel.org::akpm@linux-foundation.org:virtio-dev@lists.oasis-open.org:osalvador@suse.de:yang.zhang.wz@gmail.com:pagupta@redhat.com:riel@surriel.com:konrad.wilk@oracle.com:lcapitulino@redhat.com:wei.w.wang@intel.com:aarcange@redhat.com:pbonzini@redhat.com:dan.j.williams@intel.com:alexander.h.duyck@linux.intel.com,RULES_HIT:41:69:152:355:379:960:966:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2895:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:5007:6261:6653:6737:7576:8660:9036:9413:10004:10400:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12760:12986:13148:13230:13869: 14096:14 X-HE-Tag: corn58_58ccd3e67e337 X-Filterd-Recvd-Size: 5632 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 6 Sep 2019 14:53:30 +0000 (UTC) Received: by mail-pl1-f194.google.com with SMTP id t11so3283370plo.0 for ; Fri, 06 Sep 2019 07:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=y1UWMqHdCnekLv2dWzC/1NNH0fQiwzlC8Wszz/ILY4M=; b=Mag0NQ/BaY/evlZK7rDqVpVa4SoD20yegY2gnU1J5QJ3cQlPeV/WcjiNsMRjYj4UlO j47A4eB1Yyu6XTdDMn6pahVscdhXahzaiE5xjJhESZqtYaG0lzLUqBsiW6/Fy9NG09xg C0aLIvy/oDX59quZyauouhcpPyWzxdO81zIGT2WvnflPokAj/pLHMhQ3mj+xzsqp14Ls 7ab/Q7uvHBNht0vduEP9dnyrW2UuVdwFLJ4a6jdGsKR41YoqTUo9nnlYCGO6X4dQ9RcX uROSr2vodVMG9kRIlO3cfEccObq9fvrETCZcgM7na40pGHl+LKkghJ1Aria9vjVlnarm gZnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=y1UWMqHdCnekLv2dWzC/1NNH0fQiwzlC8Wszz/ILY4M=; b=Yscc8rHRYvmhP3oZQf0ttZVR/5xFf8fvGm0q+lsZ7W1j3s+axkNgBHkvdgzl08EKZ6 0kK/ggUQhfVos01qGKsO0Yee3Ir72PT3eSXdjffyeCWFbjSgh73ONtamGa+iqE2ft0BN gQGGjaYzz3YXvbJPMHSaiptxaA9g5g4yAphct+84ltE+c/ePj6E7KDI+JwI4xH+AZZ2C E4gX1s3sre185aoIqYMwunpVJPnYd9vvy/9r2yO8SpIdNsyukRxR2IPNtSDUZ7SllzwJ LloO0s0T8aYunhmwvWxvQcIfA1z4jkbQebeJwUtt+bOy3DuVN51rZkvGXzrXkT0nvq5G zkMA== X-Gm-Message-State: APjAAAXgrUz/UJwvrWbXrZn7/52nw9g3dO+f10B+LlmE9ee61xs4rUf7 GSPp4v61KrFXijoUZ4+yJdM= X-Google-Smtp-Source: APXvYqwXbA80G+oF/wmxkLk0iz+eT1kKPHeORFg6MaYksK5+TYFebrUDE34FVzIfkIJ4DDYs9NYZbQ== X-Received: by 2002:a17:902:a01:: with SMTP id 1mr9873072plo.278.1567781608953; Fri, 06 Sep 2019 07:53:28 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id c1sm11390068pfb.135.2019.09.06.07.53.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Sep 2019 07:53:28 -0700 (PDT) Subject: [PATCH v8 1/7] mm: Add per-cpu logic to page shuffling From: Alexander Duyck To: nitesh@redhat.com, kvm@vger.kernel.org, mst@redhat.com, david@redhat.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, mhocko@kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, virtio-dev@lists.oasis-open.org, osalvador@suse.de Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, riel@surriel.com, konrad.wilk@oracle.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com Date: Fri, 06 Sep 2019 07:53:27 -0700 Message-ID: <20190906145327.32552.39455.stgit@localhost.localdomain> In-Reply-To: <20190906145213.32552.30160.stgit@localhost.localdomain> References: <20190906145213.32552.30160.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Change the logic used to generate randomness in the suffle path so that we can avoid cache line bouncing. The previous logic was sharing the offset and entropy word between all CPUs. As such this can result in cache line bouncing and will ultimately hurt performance when enabled. To resolve this I have moved to a per-cpu logic for maintaining a unsigned long containing some amount of bits, and an offset value for which bit we can use for entropy with each call. Signed-off-by: Alexander Duyck Reviewed-by: Dan Williams --- mm/shuffle.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/mm/shuffle.c b/mm/shuffle.c index 3ce12481b1dc..9ba542ecf335 100644 --- a/mm/shuffle.c +++ b/mm/shuffle.c @@ -183,25 +183,38 @@ void __meminit __shuffle_free_memory(pg_data_t *pgdat) shuffle_zone(z); } +struct batched_bit_entropy { + unsigned long entropy_bool; + int position; +}; + +static DEFINE_PER_CPU(struct batched_bit_entropy, batched_entropy_bool); + void add_to_free_area_random(struct page *page, struct free_area *area, int migratetype) { - static u64 rand; - static u8 rand_bits; + struct batched_bit_entropy *batch; + unsigned long entropy; + int position; /* - * The lack of locking is deliberate. If 2 threads race to - * update the rand state it just adds to the entropy. + * We shouldn't need to disable IRQs as the only caller is + * __free_one_page and it should only be called with the zone lock + * held and either from IRQ context or with local IRQs disabled. */ - if (rand_bits == 0) { - rand_bits = 64; - rand = get_random_u64(); + batch = raw_cpu_ptr(&batched_entropy_bool); + position = batch->position; + + if (--position < 0) { + batch->entropy_bool = get_random_long(); + position = BITS_PER_LONG - 1; } - if (rand & 1) + batch->position = position; + entropy = batch->entropy_bool; + + if (1ul & (entropy >> position)) add_to_free_area(page, area, migratetype); else add_to_free_area_tail(page, area, migratetype); - rand_bits--; - rand >>= 1; }