From patchwork Mon Apr 15 08:41:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhenwei pi X-Patchwork-Id: 13629649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72EECC4345F for ; Mon, 15 Apr 2024 08:41:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 049336B0099; Mon, 15 Apr 2024 04:41:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F15A76B009B; Mon, 15 Apr 2024 04:41:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8F326B009C; Mon, 15 Apr 2024 04:41:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AFEF36B0099 for ; Mon, 15 Apr 2024 04:41:40 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 67867A161F for ; Mon, 15 Apr 2024 08:41:40 +0000 (UTC) X-FDA: 82011122760.30.02F5B5F Received: from mail-oo1-f52.google.com (mail-oo1-f52.google.com [209.85.161.52]) by imf30.hostedemail.com (Postfix) with ESMTP id 8291580015 for ; Mon, 15 Apr 2024 08:41:38 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=eRsKow2j; spf=pass (imf30.hostedemail.com: domain of pizhenwei@bytedance.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=pizhenwei@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713170498; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jVSyUCsRBJHReEwHJLY9AUjYGa31t9M8IXFXE6DP2gA=; b=sXBjOA2wIQRkTL7arUFOdBw41kpsMudyh0INH3EWFj9xo3v3jFT+pyWlu3F20zB2nLb1Mh ZeT9Dbg8/wRbB1hN1ZQJTbMt7qtGodjEpTFlPdBJkYyLI4p/snFqvm2C/phzJ56hzgd2sR ae6qqn++qhAWEwIDeODDP2TnJ52nm2s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713170498; a=rsa-sha256; cv=none; b=DAQtjatX4fsmhQ5irFnK28M2wrTvufq3MSWq3KFeJklfp3jg5LG+5XjHXkfn8tFD2AJmHd tdJRgadlwlr4N1B9+64nEUwe94pqm3GUJQ5JhTCTo/GmBfr3jBpH/yOdOmgmcsW7P6Z7fP fLTJJ4CDfaPmodUgJmWXv652a4v7CFQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=eRsKow2j; spf=pass (imf30.hostedemail.com: domain of pizhenwei@bytedance.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=pizhenwei@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-oo1-f52.google.com with SMTP id 006d021491bc7-5aa12c21ff4so2268219eaf.3 for ; Mon, 15 Apr 2024 01:41:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1713170497; x=1713775297; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jVSyUCsRBJHReEwHJLY9AUjYGa31t9M8IXFXE6DP2gA=; b=eRsKow2jKNSOceSp4GxCQVFGdJec9JOmbx63RuBTwFscpdswlYTN9VvvhqGBY7aUIQ L+cGdQjgsKfWLDRYNLcb4EX36xOUv02OWsFNevS/C3G9w3IR5Mpbq4YI4b/sYADX9r6q qL6CWAXyaZjNHRbJA3l1m0VEaP8j3AQuxNDZG7apCRa9MJCo4SX+2iw9Gv9VnYlavhP2 LebQDwPfZpldk74kHNflByvSJGnajF3t+7ptT626EUonBRfUFSh+flK9T+8YB+lv8yhc Wma/1blr0wU/pe/kKrMTJLjjAvHkp1xi8ED6o1ozEV3F69egbpqw4OlhIr396BZlaMsR cQ+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713170497; x=1713775297; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jVSyUCsRBJHReEwHJLY9AUjYGa31t9M8IXFXE6DP2gA=; b=Zx+vojj0prrqNTDG6NIleb/+Z1AlzWVbbUOkSh3bPuhteWGxxh9O6z2RWWzk/jzHEH KQkKqBPQINOOsrKATGZYyjLn4Gmq5Iom2T4NpoRKFtWH6ZsQ354y+T6Qa8z0FGbkN4P7 TaNejJdyyNrYgXe212fo8oXKW1hBGeOoLqE8T8yKRpwrAYxBOidny/m3ocXLpOwckkGs rOOFYKJmIU1pYYAjrEPz+bKkFCSMrZQESjK1Z4ptjIkcFWYQUhjl1MwNxiRjuV8sGnMU f1WWAh/ZJg+ofQfY07SicdmgsKBQIhoscBwRyu639OOYLzgEOQ8wj8xjWXTXoS1O1MHI IYAw== X-Forwarded-Encrypted: i=1; AJvYcCWiAwdmWZCTehG/GTUJ/V1Y+087i5+/IgCOZnQBhmOord1+RM8dfq5ZaItUyVqqcp9Jwk8gDwViU27AtyGYb1UVpZ0= X-Gm-Message-State: AOJu0YzClXyYNuTeCAn2PKMVZ25TmzyHW2grXQ7UeUg6aKcmqzAr5dYO 0UllryyddA6ag26BOQNczznRF+cdj/eGPO2dpANGcsZXmqdHyqrYSUNPgYtxVgQ= X-Google-Smtp-Source: AGHT+IFLEEPCt9VyzH/L1QCrMmvcBH5qldAmYAh+iqUObe92I/ilLOhZ5GsT6F60/+szU0KPLSFuSA== X-Received: by 2002:a05:6870:9113:b0:22f:7513:f20a with SMTP id o19-20020a056870911300b0022f7513f20amr11633580oae.55.1713170497684; Mon, 15 Apr 2024 01:41:37 -0700 (PDT) Received: from libai.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id k187-20020a636fc4000000b005d6a0b2efb3sm6575685pgc.21.2024.04.15.01.41.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 01:41:37 -0700 (PDT) From: zhenwei pi To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, virtualization@lists.linux.dev Cc: mst@redhat.com, david@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, akpm@linux-foundation.org, zhenwei pi Subject: [RFC 3/3] virtio_balloon: introduce memory scan/reclaim info Date: Mon, 15 Apr 2024 16:41:13 +0800 Message-Id: <20240415084113.1203428-4-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240415084113.1203428-1-pizhenwei@bytedance.com> References: <20240415084113.1203428-1-pizhenwei@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8291580015 X-Rspam-User: X-Stat-Signature: 8ipxryzjzn3xs4yta49doipzx9yptw9a X-Rspamd-Server: rspam03 X-HE-Tag: 1713170498-851711 X-HE-Meta: U2FsdGVkX1+ZoYe2NN0XLeK3NnzUXK20D6w6AvRRyf8KOReFP1mqUPJxcdU+bbCpGCkOl7dr1Efqw97+Z/dhkL6bZi9bPejGvvZXpBpYsRfpm8kG3PVkZCHj8GgmnRF546+pRjd5nN/+hL0jy4vjfHyiosZ4zTM4AZDrMBVfmQIbIN/D820ZUcLYmxmpuiqTPfq0vZjNElEj1Tr8Nj+RMGTiWRR/8/7lvnlD5lREp0h9/om/kgfhk12jo1npCGTIMdhGebeHq4b9nsY/fSiZu8s1CH22YBJk6U+M6hvm3xvg2w1uhxgri3QTRY1P5fHhdQmlRd/410CDz6H/sSDBazmUy2VwdZteER1C1XlEGgbHK+3qtkh3xsECtgqXRgNJQLUMvue8sRdxkk0QBTol1/HEnYT5Ucraf+qUCZaEIlA9lwbtzyxNs/D/FDa9kUJCcd0mCXMRxs6F6PCux01C3Rh7OQHJTQWbGSM7FZovNMIvrelNzCPI4e1H/084iI0dcE//QxGJYz+BkTnHmeYaPEgPH1tWFTtuNrIxjwzfm58tiWS/gEM7YEo3A7OK6DpvFsaMIN+YJhfmZ9uSS6O26Ew47+C/GkaNqa4zsIE9KIsEVzmlgUT7LyaXat7ZOBHhXp0J2NZJOJrLofhnqWAV6Pa8IXhU0iYI8x71ZMBG/kSn6+jyALrgxBDgz/oU/6fFypQIZueY17t4QtjUNDMoKMUli6pPV4xVjkrfF/DNgB1w9eESwasFNWe5/RDsm056TyLbQSHU3e1yTGpsUnUWdmJkkYKWtgZMNzUPAKmdiBguDL0ErBWTnawVXgAobf7zKxgyvUucT8mN9CtaC62TIb8wzyw9CCqP3wxeuXmsHBQIh2RdYMtSqy7Puiooa6mK/KkvAXHNPmdZXNOY6DT05NOqo+goHN8HBxhjUAGYojkaWY8oKEIANMpr2GFSkGUQnWlwaziYoRLD2EKSmJ1 qOAwyHXg 738Um/sAW9cIZpt89e02z+0gF0WRxUkL+YFF7w09KI0DaHKNxSUVD1ILHkDfKzJ6nxi5R2yEe42ZHk5xA7Eksnkhgd0KUkQTNz/XFpMTnpfNwrdDUX3bcPSo8jJG/a+lcS5TYYJQBAp7echsRS36CfpjVu2oaVEyQrESdBV6EI5LFzlzqIPxbzBrYyLQQx7ZFKKbC27JwGfk/j8kRSj1g4ZoI6DUWDaFZa/RyV82oimLJQumNoWi2N7sg2GUPuaGdFbOFwDm9b/3eo0W6md9lvPpJY0MMszXB/TK2YP+k9CC6Wi3RbgrDplLc7C+D5HS6mTfjLa864dhyK6o2uYFn9PglksSgNZFRo9zPsMfJtgvDClx3sBq9aM7td3JLe465EjZQR41hDg6xzB04VZHsDCHv3rmC/nOENV//EX6q5cTtORlHfz3Z/poMGtr9Uq7X6igLWVs1Gpbc8+c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Expose memory scan/reclaim information to the host side via virtio balloon device. Now we have a metric to analyze the memory performance: y: counter increases n: counter does not changes h: the rate of counter change is high l: the rate of counter change is low OOM: VIRTIO_BALLOON_S_OOM_KILL STALL: VIRTIO_BALLOON_S_ALLOC_STALL ASCAN: VIRTIO_BALLOON_S_SCAN_ASYNC DSCAN: VIRTIO_BALLOON_S_SCAN_DIRECT ARCLM: VIRTIO_BALLOON_S_RECLAIM_ASYNC DRCLM: VIRTIO_BALLOON_S_RECLAIM_DIRECT - OOM[y], STALL[*], ASCAN[*], DSCAN[*], ARCLM[*], DRCLM[*]: the guest runs under really critial memory pressure - OOM[n], STALL[h], ASCAN[*], DSCAN[l], ARCLM[*], DRCLM[l]: the memory allocation stalls due to cgroup, not the global memory pressure. - OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[h]: the memory allocation stalls due to global memory pressure. The performance gets hurt a lot. A high ratio between DRCLM/DSCAN shows quite effective memory reclaiming. - OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[l]: the memory allocation stalls due to global memory pressure. the ratio between DRCLM/DSCAN gets low, the guest OS is thrashing heavily, the serious case leads poor performance and difficult trouble shooting. Ex, sshd may block on memory allocation when accepting new connections, a user can't login a VM by ssh command. - OOM[n], STALL[n], ASCAN[h], DSCAN[n], ARCLM[l], DRCLM[n]: the low ratio between ARCLM/ASCAN shows that the guest tries to reclaim more memory, but it can't. Once more memory is required in future, it will struggle to reclaim memory. Signed-off-by: zhenwei pi --- drivers/virtio/virtio_balloon.c | 9 +++++++++ include/uapi/linux/virtio_balloon.h | 12 ++++++++++-- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index 4b9c9569f6e5..7b86514e99d4 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -372,6 +372,15 @@ static unsigned int update_balloon_stats(struct virtio_balloon *vb) stall += events[ALLOCSTALL_MOVABLE]; update_stat(vb, idx++, VIRTIO_BALLOON_S_ALLOC_STALL, stall); + update_stat(vb, idx++, VIRTIO_BALLOON_S_SCAN_ASYNC, + pages_to_bytes(events[PGSCAN_KSWAPD])); + update_stat(vb, idx++, VIRTIO_BALLOON_S_SCAN_DIRECT, + pages_to_bytes(events[PGSCAN_DIRECT])); + update_stat(vb, idx++, VIRTIO_BALLOON_S_RECLAIM_ASYNC, + pages_to_bytes(events[PGSTEAL_KSWAPD])); + update_stat(vb, idx++, VIRTIO_BALLOON_S_RECLAIM_DIRECT, + pages_to_bytes(events[PGSTEAL_DIRECT])); + return idx; } diff --git a/include/uapi/linux/virtio_balloon.h b/include/uapi/linux/virtio_balloon.h index 13d0c32ba27c..0875a9cccb01 100644 --- a/include/uapi/linux/virtio_balloon.h +++ b/include/uapi/linux/virtio_balloon.h @@ -73,7 +73,11 @@ struct virtio_balloon_config { #define VIRTIO_BALLOON_S_HTLB_PGFAIL 9 /* Hugetlb page allocation failures */ #define VIRTIO_BALLOON_S_OOM_KILL 10 /* OOM killer invocations */ #define VIRTIO_BALLOON_S_ALLOC_STALL 11 /* Stall count of memory allocatoin */ -#define VIRTIO_BALLOON_S_NR 12 +#define VIRTIO_BALLOON_S_SCAN_ASYNC 12 /* Amount of memory scanned asynchronously */ +#define VIRTIO_BALLOON_S_SCAN_DIRECT 13 /* Amount of memory scanned directly */ +#define VIRTIO_BALLOON_S_RECLAIM_ASYNC 14 /* Amount of memory reclaimed asynchronously */ +#define VIRTIO_BALLOON_S_RECLAIM_DIRECT 15 /* Amount of memory reclaimed directly */ +#define VIRTIO_BALLOON_S_NR 16 #define VIRTIO_BALLOON_S_NAMES_WITH_PREFIX(VIRTIO_BALLOON_S_NAMES_prefix) { \ VIRTIO_BALLOON_S_NAMES_prefix "swap-in", \ @@ -87,7 +91,11 @@ struct virtio_balloon_config { VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-allocations", \ VIRTIO_BALLOON_S_NAMES_prefix "hugetlb-failures", \ VIRTIO_BALLOON_S_NAMES_prefix "oom-kill", \ - VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall" \ + VIRTIO_BALLOON_S_NAMES_prefix "alloc-stall", \ + VIRTIO_BALLOON_S_NAMES_prefix "scan-async", \ + VIRTIO_BALLOON_S_NAMES_prefix "scan-direct", \ + VIRTIO_BALLOON_S_NAMES_prefix "reclaim-async", \ + VIRTIO_BALLOON_S_NAMES_prefix "reclaim-direct" \ } #define VIRTIO_BALLOON_S_NAMES VIRTIO_BALLOON_S_NAMES_WITH_PREFIX("")