From patchwork Thu Apr 7 10:33:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12805008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 609E8C433EF for ; Thu, 7 Apr 2022 10:37:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A82718D0005; Thu, 7 Apr 2022 06:34:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A31356B007D; Thu, 7 Apr 2022 06:34:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 904838D0005; Thu, 7 Apr 2022 06:34:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 8275D6B007B for ; Thu, 7 Apr 2022 06:34:17 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 47C25611AC for ; Thu, 7 Apr 2022 10:34:07 +0000 (UTC) X-FDA: 79329722934.03.DC478F9 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf14.hostedemail.com (Postfix) with ESMTP id F3C7D100007 for ; Thu, 7 Apr 2022 10:34:05 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id n18so4492526plg.5 for ; Thu, 07 Apr 2022 03:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=ujLNXWvcoTkgsry9MTgtB72dmND/1PFu6sESLiS4+5Q=; b=IRSMB39q0CkKZ7RhIs/Puw/gDrCfZqc84S2LziFCQ2v31cXA038xuV2TnDuQwdTqtX oQP68wBFvqS0P67Wu65Z1ImXWFUJ5J5ZmMDfviT2s2FvkOXcaaRb68QMHqSwDC9mHzXp bZfCENUVcTaBnAzz4hEQbHsNxqap60kTK+AMSq9SR1uBOqhBPoVbu1L7cUHTbQw4s+g8 Jm8KHgQHc5pS6caVI17xlHNuxOkF9d5Fmxqg/T7252I14hURjFA9vGMAYyZX1ERKOHHp AiHo3lBLT7yjAoYjQB2YZsRZCCDvQL2A1MnEJlHEC38o9O8oV4kD+NuoA+1joxlTgFx2 nrYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=ujLNXWvcoTkgsry9MTgtB72dmND/1PFu6sESLiS4+5Q=; b=W2IIEvGfVlDsxo3yPCB0+P2sHsXvj9ZyWspV1+XUSOx5bVXbITHFu/bOhjw6hjOlFY Hr3Y3Nhh8xadF2YLwx4Wg1IytWDWuwEpBg5RVyAVDWNCKjZr8dtViXbAH08yiyjXUrbG P7SDYNI/wuEN278mqY1AocVmPkYjYZrAMyPxd3nii/jqOLdtmRGADfdvLOw0MyL3etut GWJ6cZVTdZMIMEvJipRxniqFGDxK71Qv5VcUGjhKKlJKbH4HMKAQ5zTIdoa5wNVQVZSF cez1rlaLH5wgSU5T9mq1YzxK2loTeHhrs2/DlZm/mnytnEsvda3hMu0JrEllz4iLMWG7 yPoA== X-Gm-Message-State: AOAM532U42l/psIBLfblaeBrAjHIM29G5N6SL6f3TKnIObCzeC4gk1b3 XBOrDsaM1O9s9W4/5ipxOyt0Fw== X-Google-Smtp-Source: ABdhPJwoETx1Gs+umqB0Nk2NhJr5PIpA6CfK0Uc2IZapKwxyVzkhpH2VfsDLbd7RH2LZx2NVXkGLMA== X-Received: by 2002:a17:902:f2d3:b0:156:a551:fba3 with SMTP id h19-20020a170902f2d300b00156a551fba3mr13033984plc.64.1649327644604; Thu, 07 Apr 2022 03:34:04 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.250]) by smtp.gmail.com with ESMTPSA id f4-20020a056a0022c400b004fb2292cd74sm22601362pfj.206.2022.04.07.03.34.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Apr 2022 03:34:04 -0700 (PDT) From: Qi Zheng To: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, zhouchengming@bytedance.com, songmuchun@bytedance.com, Qi Zheng Subject: [PATCH] percpu_ref: call wake_up_all() after percpu_ref_put() completes Date: Thu, 7 Apr 2022 18:33:35 +0800 Message-Id: <20220407103335.36885-1-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: xxqwu95bntfcqjc9edih46met5u9aa55 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=IRSMB39q; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: F3C7D100007 X-HE-Tag: 1649327645-255850 X-Bogosity: Ham, tests=bogofilter, spamicity=0.002195, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the percpu_ref_call_confirm_rcu(), we call the wake_up_all() before calling percpu_ref_put(), which will cause the value of percpu_ref to be unstable when percpu_ref_switch_to_atomic_sync() returns. CPU0 CPU1 percpu_ref_switch_to_atomic_sync(&ref) --> percpu_ref_switch_to_atomic(&ref) --> percpu_ref_get(ref); /* put after confirmation */ call_rcu(&ref->data->rcu, percpu_ref_switch_to_atomic_rcu); percpu_ref_switch_to_atomic_rcu --> percpu_ref_call_confirm_rcu --> data->confirm_switch = NULL; wake_up_all(&percpu_ref_switch_waitq); /* here waiting to wake up */ wait_event(percpu_ref_switch_waitq, !ref->data->confirm_switch); (A)percpu_ref_put(ref); /* The value of &ref is unstable! */ percpu_ref_is_zero(&ref) (B)percpu_ref_put(ref); As shown above, assuming that the counts on each cpu add up to 0 before calling percpu_ref_switch_to_atomic_sync(), we expect that after switching to atomic mode, percpu_ref_is_zero() can return true. But actually it will return different values in the two cases of A and B, which is not what we expected. Maybe the original purpose of percpu_ref_switch_to_atomic_sync() is just to ensure that the conversion to atomic mode is completed, but it should not return with an extra reference count. Calling wake_up_all() after percpu_ref_put() ensures that the value of percpu_ref is stable after percpu_ref_switch_to_atomic_sync() returns. So just do it. Signed-off-by: Qi Zheng Acked-by: Dennis Zhou Reviewed-by: Ming Lei Reviewed-by: Muchun Song --- lib/percpu-refcount.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index af9302141bcf..b11b4152c8cd 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -154,13 +154,14 @@ static void percpu_ref_call_confirm_rcu(struct rcu_head *rcu) data->confirm_switch(ref); data->confirm_switch = NULL; - wake_up_all(&percpu_ref_switch_waitq); if (!data->allow_reinit) __percpu_ref_exit(ref); /* drop ref from percpu_ref_switch_to_atomic() */ percpu_ref_put(ref); + + wake_up_all(&percpu_ref_switch_waitq); } static void percpu_ref_switch_to_atomic_rcu(struct rcu_head *rcu)