From patchwork Wed Jun 16 03:21:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Lutomirski X-Patchwork-Id: 12323805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96CC5C48BE8 for ; Wed, 16 Jun 2021 03:21:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 265AE613E9 for ; Wed, 16 Jun 2021 03:21:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 265AE613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 32C406B006E; Tue, 15 Jun 2021 23:21:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 301FF6B0070; Tue, 15 Jun 2021 23:21:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A1FE6B0071; Tue, 15 Jun 2021 23:21:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id DC7FF6B006E for ; Tue, 15 Jun 2021 23:21:17 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8217E181AC9B6 for ; Wed, 16 Jun 2021 03:21:17 +0000 (UTC) X-FDA: 78258136194.33.ED203BD Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf08.hostedemail.com (Postfix) with ESMTP id 69B40801912B for ; Wed, 16 Jun 2021 03:21:07 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 299A6613C2; Wed, 16 Jun 2021 03:21:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623813676; bh=H/1rsxHk8ldroOc2+bi7w574v2BCSyir0ZtMa99Tvns=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ak4D+cP32X7Yl7slNXj0+783UpJ9FByLmWHAKV/geREVCHo/6YJba1kXFom/wdPmC DbRSB/5eUwVgk5DSI9wOAdEn/iPagIJbKcwAeJPMsgt6DzFRN8NvCSyOK0kGqZGO7C 3hF66t74y2hPfm29zLySP0LlpB4yhUc6JBd/+ML4aK/5GMvclAIShjiQrwOds4c9Vn 0a4onM2CsctRRCWcKYItD5BKM2WqZmgvmNUzs+62JyUTz3vQOF6HdYdAXFRpw/VM+c soZLf4m/ToH+miHTFXdcr0n22CGthb/ip7ngvOKBLpQi0OO+6dK6+mNk491T1uCQm6 hQ6ngyb4wJBTA== From: Andy Lutomirski To: x86@kernel.org Cc: Dave Hansen , LKML , linux-mm@kvack.org, Andrew Morton , Andy Lutomirski , Mathieu Desnoyers , Nicholas Piggin , Peter Zijlstra Subject: [PATCH 1/8] membarrier: Document why membarrier() works Date: Tue, 15 Jun 2021 20:21:06 -0700 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Ak4D+cP3; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of luto@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=luto@kernel.org X-Stat-Signature: 6xgawzx6swfyxi174zreptnwb6tifazg X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 69B40801912B X-HE-Tag: 1623813667-630435 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We had a nice comment at the top of membarrier.c explaining why membarrier worked in a handful of scenarios, but that consisted more of a list of things not to forget than an actual description of the algorithm and why it should be expected to work. Add a comment explaining my understanding of the algorithm. This exposes a couple of implementation issues that I will hopefully fix up in subsequent patches. Cc: Mathieu Desnoyers Cc: Nicholas Piggin Cc: Peter Zijlstra Signed-off-by: Andy Lutomirski --- kernel/sched/membarrier.c | 55 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c index b5add64d9698..3173b063d358 100644 --- a/kernel/sched/membarrier.c +++ b/kernel/sched/membarrier.c @@ -7,6 +7,61 @@ #include "sched.h" /* + * The basic principle behind the regular memory barrier mode of membarrier() + * is as follows. For each CPU, membarrier() operates in one of two + * modes. Either it sends an IPI or it does not. If membarrier() sends an + * IPI, then we have the following sequence of events: + * + * 1. membarrier() does smp_mb(). + * 2. membarrier() does a store (the IPI request payload) that is observed by + * the target CPU. + * 3. The target CPU does smp_mb(). + * 4. The target CPU does a store (the completion indication) that is observed + * by membarrier()'s wait-for-IPIs-to-finish request. + * 5. membarrier() does smp_mb(). + * + * So all pre-membarrier() local accesses are visible after the IPI on the + * target CPU and all pre-IPI remote accesses are visible after + * membarrier(). IOW membarrier() has synchronized both ways with the target + * CPU. + * + * (This has the caveat that membarrier() does not interrupt the CPU that it's + * running on at the time it sends the IPIs. However, if that is the CPU on + * which membarrier() starts and/or finishes, membarrier() does smp_mb() and, + * if not, then membarrier() scheduled, and scheduling had better include a + * full barrier somewhere for basic correctness regardless of membarrier.) + * + * If membarrier() does not send an IPI, this means that membarrier() reads + * cpu_rq(cpu)->curr->mm and that the result is not equal to the target + * mm. Let's assume for now that tasks never change their mm field. The + * sequence of events is: + * + * 1. Target CPU switches away from the target mm (or goes lazy or has never + * run the target mm in the first place). This involves smp_mb() followed + * by a write to cpu_rq(cpu)->curr. + * 2. membarrier() does smp_mb(). (This is NOT synchronized with any action + * done by the target.) + * 3. membarrier() observes the value written in step 1 and does *not* observe + * the value written in step 5. + * 4. membarrier() does smp_mb(). + * 5. Target CPU switches back to the target mm and writes to + * cpu_rq(cpu)->curr. (This is NOT synchronized with any action on + * membarrier()'s part.) + * 6. Target CPU executes smp_mb() + * + * All pre-schedule accesses on the remote CPU are visible after membarrier() + * because they all precede the target's write in step 1 and are synchronized + * to the local CPU by steps 3 and 4. All pre-membarrier() accesses on the + * local CPU are visible on the remote CPU after scheduling because they + * happen before the smp_mb(); read in steps 2 and 3 and that read preceeds + * the target's smp_mb() in step 6. + * + * However, tasks can change their ->mm, e.g., via kthread_use_mm(). So + * tasks that switch their ->mm must follow the same rules as the scheduler + * changing rq->curr, and the membarrier() code needs to do both dereferences + * carefully. + * + * * For documentation purposes, here are some membarrier ordering * scenarios to keep in mind: *