From patchwork Fri Oct 8 16:19:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Saenz Julienne X-Patchwork-Id: 12545685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16B20C433FE for ; Fri, 8 Oct 2021 16:19:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AB04F610D1 for ; Fri, 8 Oct 2021 16:19:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AB04F610D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C43336B0071; Fri, 8 Oct 2021 12:19:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BF38F6B0072; Fri, 8 Oct 2021 12:19:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABAEC6B0073; Fri, 8 Oct 2021 12:19:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0226.hostedemail.com [216.40.44.226]) by kanga.kvack.org (Postfix) with ESMTP id 9CD316B0071 for ; Fri, 8 Oct 2021 12:19:38 -0400 (EDT) Received: from smtpin35.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 558BF8249980 for ; Fri, 8 Oct 2021 16:19:38 +0000 (UTC) X-FDA: 78673780836.35.417E506 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id DAC323000ADC for ; Fri, 8 Oct 2021 16:19:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633709977; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3cEPXatX2ZSJNov0QZ84u3WSw59F+CiwA/vBLiu9m6E=; b=Bha2vO7Jg5sfSPQMawJPDxMwGCwko4Nlu9e3qi8lOItmgtqanurZ5VgOxTGekhKFnxBEq4 +7sgmN9xrUmeIKRRWwKsK1RJxwfup2g9Pgn5aDpVuHEEN3CsJnnWjeUVhlhYve2ptSbLsD QjB7FSJosYSiycGjgTaGCtBval03Pw4= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-578-g4Px7hLwMzWCQP6b7TqiUQ-1; Fri, 08 Oct 2021 12:19:34 -0400 X-MC-Unique: g4Px7hLwMzWCQP6b7TqiUQ-1 Received: by mail-wr1-f71.google.com with SMTP id d13-20020adfa34d000000b00160aa1cc5f1so7714021wrb.14 for ; Fri, 08 Oct 2021 09:19:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=3cEPXatX2ZSJNov0QZ84u3WSw59F+CiwA/vBLiu9m6E=; b=OHSeiAsm9HMG9430CfcG3aE5K7Iwcq8NYlqikre1IYKrP19sDTLcRlDPwf6n0Krz3j p8Yw+of9aGOFgVDRLZVfPXsnWIgr0zYDqfLwvGtPR3C7MM/0YfFdBoV2gsGAz0KlNewJ ft0jWYsfSrs9cZQDQpa8s5pNIafOzZDWwmeHLXsMtXPTuNEopBvZHTE7pkxB+/negI9y xHy4r7h/xnLI8fN97SdtAuLysgoJkfR5NK05i+P3sneFKryzaWR6loOEM6wfsSqDWH4J wg9iCUzge7TCht4ya7ZUdyBtj5xaRLQwOtS/QBRpHQ4lveodk7poR+SPe8Rao71vEDtk 05Qw== X-Gm-Message-State: AOAM531YY95DMxaDg0sobbGdhky69G/hFDBEXNFtXP8uQTrASsfkri+t 3MPHlu6EClHHu5+ENfImkQ9UjKyW5g46buxDN3it6GbFp4pcg/LAhL6Or45K0paGm3mzXbzlX5I 025owwrzlGj8= X-Received: by 2002:a1c:7d91:: with SMTP id y139mr4493841wmc.57.1633709973167; Fri, 08 Oct 2021 09:19:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwKKijO+74izv+qNkXyDuYFKonDBhllUKwll6+hoH5cpRMPBMvTIj5SMLDKxABYXEH63mxR+A== X-Received: by 2002:a1c:7d91:: with SMTP id y139mr4493808wmc.57.1633709972941; Fri, 08 Oct 2021 09:19:32 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id f184sm2901753wmf.22.2021.10.08.09.19.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Oct 2021 09:19:32 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, frederic@kernel.org, tglx@linutronix.de, peterz@infradead.org, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, linux-rt-users@vger.kernel.org, vbabka@suse.cz, cl@linux.com, paulmck@kernel.org, ppandit@redhat.com, Nicolas Saenz Julienne Subject: [RFC 0/3] mm/page_alloc: Remote per-cpu lists drain support Date: Fri, 8 Oct 2021 18:19:19 +0200 Message-Id: <20211008161922.942459-1-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: DAC323000ADC X-Stat-Signature: 989m1hqnn43a4m3zgi34bxfez37eaeju Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Bha2vO7J; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf08.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=nsaenzju@redhat.com X-HE-Tag: 1633709977-340093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This series replaces mm/page_alloc's per-cpu lists drain mechanism in order for it to be able to be run remotely. Currently, only a local CPU is permitted to change its per-cpu lists, and it's expected to do so, on-demand, whenever a process demands it (by means of queueing a drain task on the local CPU). Most systems will handle this promptly, but it'll cause problems for NOHZ_FULL CPUs that can't take any sort of interruption without breaking their functional guarantees (latency, bandwidth, etc...). Having a way for these processes to remotely drain the lists themselves will make co-existing with isolated CPUs possible, and comes with minimal performance[1]/memory cost to other users. The new algorithm will atomically switch the pointer to the per-cpu lists and use RCU to make sure it's not being used before draining them. I'm interested in an sort of feedback, but especially validating that the approach is acceptable, and any tests/benchmarks you'd like to see run against it. For now, I've been testing this successfully on both arm64 and x86_64 systems while forcing high memory pressure (i.e. forcing the page_alloc's slow path). Patches 1-2 serve as cleanups/preparation to make patch 3 easier to follow. Here's my previous attempt at fixing this: https://lkml.org/lkml/2021/9/21/599 [1] Proper performance numbers will be provided if the approach is deemed acceptable. That said, mm/page_alloc.c's fast paths only grow by an extra pointer indirection and a compiler barrier, which I think is unlikely to be measurable. --- Nicolas Saenz Julienne (3): mm/page_alloc: Simplify __rmqueue_pcplist()'s arguments mm/page_alloc: Access lists in 'struct per_cpu_pages' indirectly mm/page_alloc: Add remote draining support to per-cpu lists include/linux/mmzone.h | 24 +++++- mm/page_alloc.c | 173 +++++++++++++++++++++-------------------- mm/vmstat.c | 6 +- 3 files changed, 114 insertions(+), 89 deletions(-)