Message ID | 20211103161020.26714-1-lukasz.luba@arm.com (mailing list archive) |
---|---|
Headers | show |
Series | Refactor thermal pressure update to avoid code duplication | expand |
Hi Lukasz, On 11/3/21 11:10 AM, Lukasz Luba wrote: > Hi all, > > This patch set v3 aims to refactor the thermal pressure update > code. There are already two clients which do similar thing: > convert the capped frequency value into the capacity of > affected CPU and call the 'set' function to store the > reduced capacity into the per-cpu variable. > There might be more than two of these users. In near future > it will be scmi-cpufreq driver, which receives notification > from FW about reduced frequency due to thermal. Other vendors > might follow. Let's avoid code duplication and potential > conversion bugs. Move the conversion code into the arch_topology.c > where the capacity calculation setup code and thermal pressure sit. > > Apart from that $subject patches, there is one patch (3/5) which fixes > issue in qcom-cpufreq-hw.c when the thermal pressure is not > updated for offline CPUs. It's similar fix that has been merged > recently for cpufreq_cooling.c: > 2ad8ccc17d1e4270cf65a3f2 > > Changes: > v3: > - added warning and check if provided capped frequency is lower than > max (Viresh) > - removed check for empty cpu mask (Viresh) > - replaced tabs with spaces in the doxygen comment (Viresh) > - renamed {arch|topology}_thermal_pressure_update() to > {arch|topology}_update_thermal_pressure() so it's align with scheme (Dietmar) > - added info about MHz in freq_factor into patch description (Dietmar) > v2 [2]: > - added Reviewed-by from Thara for patch 3/5 > - changed the doxygen comment and used mult_frac() > according to Thara's suggestion in patch 1/5 > v1 -> [1] > > Regards, > Lukasz Luba > > [1] https://lore.kernel.org/linux-pm/20211007080729.8262-1-lukasz.luba@arm.com/ > [2] https://lore.kernel.org/linux-pm/20211015144550.23719-1-lukasz.luba@arm.com/ > > Lukasz Luba (5): > arch_topology: Introduce thermal pressure update function > thermal: cpufreq_cooling: Use new thermal pressure update function > cpufreq: qcom-cpufreq-hw: Update offline CPUs per-cpu thermal pressure > cpufreq: qcom-cpufreq-hw: Use new thermal pressure update function > arch_topology: Remove unused topology_set_thermal_pressure() and > related > > arch/arm/include/asm/topology.h | 2 +- > arch/arm64/include/asm/topology.h | 2 +- > drivers/base/arch_topology.c | 36 +++++++++++++++++++++++++++---- > drivers/cpufreq/qcom-cpufreq-hw.c | 14 +++++------- > drivers/thermal/cpufreq_cooling.c | 6 +----- > include/linux/arch_topology.h | 4 ++-- > include/linux/sched/topology.h | 6 +++--- > init/Kconfig | 2 +- > 8 files changed, 46 insertions(+), 26 deletions(-) > I've been testing this patchset on the Lenovo Yoga C630, and today while compiling alacritty and running an apt-get full-upgrade, I found the following in dmesg output: [ 194.343903] ------------[ cut here ]------------ [ 194.343912] WARNING: CPU: 4 PID: 192 at drivers/base/arch_topology.c:188 topology_update_thermal_pressure+0xe4/0x100 [ 194.343928] Modules linked in: aes_ce_ccm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_userspace lz4 lz4_compress zram zsmalloc q6asm_dai q6routing q6afe_dai q6adm q6asm q6afe q6dsp_common snd_soc_wsa881x q6core regmap_sdw snd_soc_wcd934x gpio_wcd934x soundwire_qcom snd_soc_wcd_mbhc wcd934x regmap_slimbus uvcvideo videobuf2_vmalloc venus_enc venus_dec videobuf2_dma_contig videobuf2_memops qrtr_smd fastrpc apr binfmt_misc nls_ascii nls_cp437 vfat fat snd_soc_sdm845 snd_soc_rt5663 snd_soc_qcom_common pm8941_pwrkey joydev snd_soc_rl6231 aes_ce_blk qcom_spmi_adc5 soundwire_bus qcom_vadc_common crypto_simd qcom_spmi_temp_alarm snd_soc_core cryptd hci_uart snd_compress btqca industrialio aes_ce_cipher snd_pcm_dmaengine btrtl crct10dif_ce btbcm ghash_ce snd_pcm btintel gf128mul sha2_ce snd_timer venus_core bluetooth snd v4l2_mem2mem sha256_arm64 videobuf2_v4l2 videobuf2_common soundcore [ 194.344007] sha1_ce videodev ecdh_generic ecc mc ath10k_snoc ath10k_core hid_multitouch ath mac80211 qcom_rng libarc4 qcom_q6v5_mss cfg80211 sg rfkill qcom_q6v5_pas qcom_pil_info slim_qcom_ngd_ctrl qcom_wdt pdr_interface qcom_q6v5 evdev rmtfs_mem slimbus qcom_sysmon fuse configfs qrtr ip_tables x_tables autofs4 ext4 mbcache jbd2 panel_simple rtc_pm8xxx msm llcc_qcom ocmem gpu_sched ti_sn65dsi86 drm_dp_aux_bus drm_kms_helper drm camcc_sdm845 ipa qcom_common qmi_helpers mdt_loader gpio_keys pwm_bl [ 194.344056] CPU: 4 PID: 192 Comm: kworker/4:1H Not tainted 5.15.0 #9 [ 194.344060] Hardware name: LENOVO 81JL/LNVNB161216, BIOS 9UCN33WW(V2.06) 06/ 4/2019 [ 194.344062] Workqueue: events_highpri qcom_lmh_dcvs_poll [ 194.344068] pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 194.344070] pc : topology_update_thermal_pressure+0xe4/0x100 [ 194.344073] lr : topology_update_thermal_pressure+0x30/0x100 [ 194.344075] sp : ffff800014043d10 [ 194.344076] x29: ffff800014043d10 x28: 0000000000000000 x27: 0000000000000000 [ 194.344080] x26: ffff759c40b66974 x25: ffff759db37a2405 x24: ffff759c40e83358 [ 194.344084] x23: 0000000000000000 x22: ffffb2699eafe1d8 x21: 00000000002d1e00 [ 194.344087] x20: ffff759c49f5bc20 x19: 000000000000b8cc x18: 0000000000000000 [ 194.344090] x17: 2f756e672d78756e x16: ffffb2699d7d83d0 x15: 0000000000000000 [ 194.344093] x14: 0000000000000000 x13: 0000000000000030 x12: 0101010101010101 [ 194.344096] x11: 7f7f7f7f7f7f7f7f x10: feff68716f676668 x9 : ffffb2699dd66a58 [ 194.344099] x8 : fefefefefefefeff x7 : 000000000000000f x6 : 0000000000000002 [ 194.344102] x5 : ffffc33415124000 x4 : 0000000000000400 x3 : 0000000000000b19 [ 194.344105] x2 : 0000000000000b8c x1 : ffffb2699e678f40 x0 : ffffb2699e678f48 [ 194.344108] Call trace: [ 194.344110] topology_update_thermal_pressure+0xe4/0x100 [ 194.344113] qcom_lmh_dcvs_notify+0xc8/0x160 [ 194.344115] qcom_lmh_dcvs_poll+0x20/0x2c [ 194.344116] process_one_work+0x1f4/0x490 [ 194.344120] worker_thread+0x188/0x504 [ 194.344121] kthread+0x12c/0x140 [ 194.344125] ret_from_fork+0x10/0x20 [ 194.344128] ---[ end trace bd0039c4fb892d5b ]--- [ 194.344170] ------------[ cut here ]------------ [ 194.344171] WARNING: CPU: 4 PID: 161 at drivers/base/arch_topology.c:188 topology_update_thermal_pressure+0xe4/0x100 [ 194.344175] Modules linked in: aes_ce_ccm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_userspace lz4 lz4_compress zram zsmalloc q6asm_dai q6routing q6afe_dai q6adm q6asm q6afe q6dsp_common snd_soc_wsa881x q6core regmap_sdw snd_soc_wcd934x gpio_wcd934x soundwire_qcom snd_soc_wcd_mbhc wcd934x regmap_slimbus uvcvideo videobuf2_vmalloc venus_enc venus_dec videobuf2_dma_contig videobuf2_memops qrtr_smd fastrpc apr binfmt_misc nls_ascii nls_cp437 vfat fat snd_soc_sdm845 snd_soc_rt5663 snd_soc_qcom_common pm8941_pwrkey joydev snd_soc_rl6231 aes_ce_blk qcom_spmi_adc5 soundwire_bus qcom_vadc_common crypto_simd qcom_spmi_temp_alarm snd_soc_core cryptd hci_uart snd_compress btqca industrialio aes_ce_cipher snd_pcm_dmaengine btrtl crct10dif_ce btbcm ghash_ce snd_pcm btintel gf128mul sha2_ce snd_timer venus_core bluetooth snd v4l2_mem2mem sha256_arm64 videobuf2_v4l2 videobuf2_common soundcore [ 194.344225] sha1_ce videodev ecdh_generic ecc mc ath10k_snoc ath10k_core hid_multitouch ath mac80211 qcom_rng libarc4 qcom_q6v5_mss cfg80211 sg rfkill qcom_q6v5_pas qcom_pil_info slim_qcom_ngd_ctrl qcom_wdt pdr_interface qcom_q6v5 evdev rmtfs_mem slimbus qcom_sysmon fuse configfs qrtr ip_tables x_tables autofs4 ext4 mbcache jbd2 panel_simple rtc_pm8xxx msm llcc_qcom ocmem gpu_sched ti_sn65dsi86 drm_dp_aux_bus drm_kms_helper drm camcc_sdm845 ipa qcom_common qmi_helpers mdt_loader gpio_keys pwm_bl [ 194.344256] CPU: 4 PID: 161 Comm: irq/159-dcvsh-i Tainted: G W 5.15.0 #9 [ 194.344259] Hardware name: LENOVO 81JL/LNVNB161216, BIOS 9UCN33WW(V2.06) 06/ 4/2019 [ 194.344260] pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 194.344262] pc : topology_update_thermal_pressure+0xe4/0x100 [ 194.344264] lr : topology_update_thermal_pressure+0x30/0x100 [ 194.344266] sp : ffff800011e93cf0 [ 194.344267] x29: ffff800011e93cf0 x28: 0000000000000000 x27: 0000000000000000 [ 194.344270] x26: ffffb2699d4d0a80 x25: ffffb2699d4d0b60 x24: ffff759c40caec00 [ 194.344273] x23: ffff759c49f02c40 x22: ffffb2699eafe1d8 x21: 00000000002d1e00 [ 194.344276] x20: ffff759c49f5bc20 x19: 000000000000b8cc x18: 000000000000000e [ 194.344280] x17: 0000000000000001 x16: 0000000000000019 x15: 0000000000000033 [ 194.344283] x14: 000000000000004c x13: 0000000000000068 x12: 0000000000000040 [ 194.344286] x11: ffff759c4049f478 x10: ffff759c4049f47a x9 : ffffb2699dd66a58 [ 194.344288] x8 : ffff759c40401dd0 x7 : 0000000000000000 x6 : 0000000000000002 [ 194.344291] x5 : ffffc33415124000 x4 : 0000000000000400 x3 : 0000000000000b19 [ 194.344294] x2 : 0000000000000b8c x1 : ffffb2699e678f40 x0 : ffffb2699e678f48 [ 194.344297] Call trace: [ 194.344298] topology_update_thermal_pressure+0xe4/0x100 [ 194.344300] qcom_lmh_dcvs_notify+0xc8/0x160 [ 194.344302] qcom_lmh_dcvs_handle_irq+0x30/0x44 [ 194.344304] irq_thread_fn+0x38/0xb0 [ 194.344307] irq_thread+0x194/0x3b0 [ 194.344308] kthread+0x12c/0x140 [ 194.344311] ret_from_fork+0x10/0x20 [ 194.344313] ---[ end trace bd0039c4fb892d5c ]--- [ 194.405913] ------------[ cut here ]------------ [ 194.405923] WARNING: CPU: 4 PID: 192 at drivers/base/arch_topology.c:188 topology_update_thermal_pressure+0xe4/0x100 [ 194.405939] Modules linked in: aes_ce_ccm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_userspace lz4 lz4_compress zram zsmalloc q6asm_dai q6routing q6afe_dai q6adm q6asm q6afe q6dsp_common snd_soc_wsa881x q6core regmap_sdw snd_soc_wcd934x gpio_wcd934x soundwire_qcom snd_soc_wcd_mbhc wcd934x regmap_slimbus uvcvideo videobuf2_vmalloc venus_enc venus_dec videobuf2_dma_contig videobuf2_memops qrtr_smd fastrpc apr binfmt_misc nls_ascii nls_cp437 vfat fat snd_soc_sdm845 snd_soc_rt5663 snd_soc_qcom_common pm8941_pwrkey joydev snd_soc_rl6231 aes_ce_blk qcom_spmi_adc5 soundwire_bus qcom_vadc_common crypto_simd qcom_spmi_temp_alarm snd_soc_core cryptd hci_uart snd_compress btqca industrialio aes_ce_cipher snd_pcm_dmaengine btrtl crct10dif_ce btbcm ghash_ce snd_pcm btintel gf128mul sha2_ce snd_timer venus_core bluetooth snd v4l2_mem2mem sha256_arm64 videobuf2_v4l2 videobuf2_common soundcore [ 194.406038] sha1_ce videodev ecdh_generic ecc mc ath10k_snoc ath10k_core hid_multitouch ath mac80211 qcom_rng libarc4 qcom_q6v5_mss cfg80211 sg rfkill qcom_q6v5_pas qcom_pil_info slim_qcom_ngd_ctrl qcom_wdt pdr_interface qcom_q6v5 evdev rmtfs_mem slimbus qcom_sysmon fuse configfs qrtr ip_tables x_tables autofs4 ext4 mbcache jbd2 panel_simple rtc_pm8xxx msm llcc_qcom ocmem gpu_sched ti_sn65dsi86 drm_dp_aux_bus drm_kms_helper drm camcc_sdm845 ipa qcom_common qmi_helpers mdt_loader gpio_keys pwm_bl [ 194.406089] CPU: 4 PID: 192 Comm: kworker/4:1H Tainted: G W 5.15.0 #9 [ 194.406092] Hardware name: LENOVO 81JL/LNVNB161216, BIOS 9UCN33WW(V2.06) 06/ 4/2019 [ 194.406094] Workqueue: events_highpri qcom_lmh_dcvs_poll [ 194.406101] pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 194.406104] pc : topology_update_thermal_pressure+0xe4/0x100 [ 194.406107] lr : topology_update_thermal_pressure+0x30/0x100 [ 194.406109] sp : ffff800014043d10 [ 194.406111] x29: ffff800014043d10 x28: 0000000000000000 x27: 0000000000000000 [ 194.406114] x26: ffff759c40b66974 x25: ffff759db37a2405 x24: ffff759c40e83358 [ 194.406117] x23: 0000000000000000 x22: ffffb2699eafe1d8 x21: 00000000002d1e00 [ 194.406120] x20: ffff759c49f5bc20 x19: 000000000000b8cc x18: 0000000000000000 [ 194.406123] x17: ffffc33415124000 x16: ffff800010024000 x15: 0000000000004000 [ 194.406126] x14: 0000000000000000 x13: 0000000000000030 x12: 0101010101010101 [ 194.406129] x11: 7f7f7f7f7f7f7f7f x10: feff68716f676668 x9 : ffffb2699dd66a58 [ 194.406132] x8 : fefefefefefefeff x7 : 000000000000000f x6 : 0000000000000002 [ 194.406135] x5 : ffffc33415124000 x4 : 0000000000000400 x3 : 0000000000000b19 [ 194.406138] x2 : 0000000000000b8c x1 : ffffb2699e678f40 x0 : ffffb2699e678f48 [ 194.406141] Call trace: [ 194.406143] topology_update_thermal_pressure+0xe4/0x100 [ 194.406146] qcom_lmh_dcvs_notify+0xc8/0x160 [ 194.406148] qcom_lmh_dcvs_poll+0x20/0x2c [ 194.406149] process_one_work+0x1f4/0x490 [ 194.406153] worker_thread+0x188/0x504 [ 194.406154] kthread+0x12c/0x140 [ 194.406158] ret_from_fork+0x10/0x20 [ 194.406162] ---[ end trace bd0039c4fb892d5d ]---
Hi Steev, On 11/5/21 3:39 PM, Steev Klimaszewski wrote: > Hi Lukasz, > [snip] > I've been testing this patchset on the Lenovo Yoga C630, and today while > compiling alacritty and running an apt-get full-upgrade, I found the > following in dmesg output: Thank you for testing and sending feedback! Are you using a mainline kernel or you applied on some vendor production kernel this patch set? I need to exclude a different code base from the equation, especially to the arch_topology.c init code. > > [ 194.343903] ------------[ cut here ]------------ > [ 194.343912] WARNING: CPU: 4 PID: 192 at > drivers/base/arch_topology.c:188 > topology_update_thermal_pressure+0xe4/0x100 > [ 194.343928] Modules linked in: aes_ce_ccm snd_seq_dummy snd_hrtimer > snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep > cpufreq_ondemand cpufreq_conservative cpufreq_powersave > cpufreq_userspace lz4 lz4_compress zram zsmalloc q6asm_dai q6routing > q6afe_dai q6adm q6asm q6afe q6dsp_common snd_soc_wsa881x q6core > regmap_sdw snd_soc_wcd934x gpio_wcd934x soundwire_qcom snd_soc_wcd_mbhc > wcd934x regmap_slimbus uvcvideo videobuf2_vmalloc venus_enc venus_dec > videobuf2_dma_contig videobuf2_memops qrtr_smd fastrpc apr binfmt_misc > nls_ascii nls_cp437 vfat fat snd_soc_sdm845 snd_soc_rt5663 > snd_soc_qcom_common pm8941_pwrkey joydev snd_soc_rl6231 aes_ce_blk > qcom_spmi_adc5 soundwire_bus qcom_vadc_common crypto_simd > qcom_spmi_temp_alarm snd_soc_core cryptd hci_uart snd_compress btqca > industrialio aes_ce_cipher snd_pcm_dmaengine btrtl crct10dif_ce btbcm > ghash_ce snd_pcm btintel gf128mul sha2_ce snd_timer venus_core bluetooth > snd v4l2_mem2mem sha256_arm64 videobuf2_v4l2 videobuf2_common soundcore > [ 194.344007] sha1_ce videodev ecdh_generic ecc mc ath10k_snoc > ath10k_core hid_multitouch ath mac80211 qcom_rng libarc4 qcom_q6v5_mss > cfg80211 sg rfkill qcom_q6v5_pas qcom_pil_info slim_qcom_ngd_ctrl > qcom_wdt pdr_interface qcom_q6v5 evdev rmtfs_mem slimbus qcom_sysmon > fuse configfs qrtr ip_tables x_tables autofs4 ext4 mbcache jbd2 > panel_simple rtc_pm8xxx msm llcc_qcom ocmem gpu_sched ti_sn65dsi86 > drm_dp_aux_bus drm_kms_helper drm camcc_sdm845 ipa qcom_common > qmi_helpers mdt_loader gpio_keys pwm_bl > [ 194.344056] CPU: 4 PID: 192 Comm: kworker/4:1H Not tainted 5.15.0 #9 > [ 194.344060] Hardware name: LENOVO 81JL/LNVNB161216, BIOS > 9UCN33WW(V2.06) 06/ 4/2019 > [ 194.344062] Workqueue: events_highpri qcom_lmh_dcvs_poll > [ 194.344068] pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS > BTYPE=--) > [ 194.344070] pc : topology_update_thermal_pressure+0xe4/0x100 > [ 194.344073] lr : topology_update_thermal_pressure+0x30/0x100 > [ 194.344075] sp : ffff800014043d10 > [ 194.344076] x29: ffff800014043d10 x28: 0000000000000000 x27: > 0000000000000000 > [ 194.344080] x26: ffff759c40b66974 x25: ffff759db37a2405 x24: > ffff759c40e83358 > [ 194.344084] x23: 0000000000000000 x22: ffffb2699eafe1d8 x21: > 00000000002d1e00 > [ 194.344087] x20: ffff759c49f5bc20 x19: 000000000000b8cc x18: > 0000000000000000 > [ 194.344090] x17: 2f756e672d78756e x16: ffffb2699d7d83d0 x15: > 0000000000000000 > [ 194.344093] x14: 0000000000000000 x13: 0000000000000030 x12: > 0101010101010101 > [ 194.344096] x11: 7f7f7f7f7f7f7f7f x10: feff68716f676668 x9 : > ffffb2699dd66a58 > [ 194.344099] x8 : fefefefefefefeff x7 : 000000000000000f x6 : > 0000000000000002 > [ 194.344102] x5 : ffffc33415124000 x4 : 0000000000000400 x3 : > 0000000000000b19 > [ 194.344105] x2 : 0000000000000b8c x1 : ffffb2699e678f40 x0 : > ffffb2699e678f48 > [ 194.344108] Call trace: > [ 194.344110] topology_update_thermal_pressure+0xe4/0x100 > [ 194.344113] qcom_lmh_dcvs_notify+0xc8/0x160 > [ 194.344115] qcom_lmh_dcvs_poll+0x20/0x2c > [ 194.344116] process_one_work+0x1f4/0x490 > [ 194.344120] worker_thread+0x188/0x504 > [ 194.344121] kthread+0x12c/0x140 > [ 194.344125] ret_from_fork+0x10/0x20 > [ 194.344128] ---[ end trace bd0039c4fb892d5b ]--- [snip] That's interesting why we hit this. I should have added info about those two values, which are compared. Could you make this change and try it again, please? We would know the problematic values, which triggered this. ---------------------8<----------------------------------- diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index db18d79065fe..0d8db0927041 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -185,8 +185,11 @@ void topology_update_thermal_pressure(const struct cpumask *cpus, /* Convert to MHz scale which is used in 'freq_factor' */ capped_freq /= 1000; - if (WARN_ON(max_freq < capped_freq)) + if (max_freq < capped_freq) { + pr_warn("THERMAL_PRESSURE: max_freq (%lu) < capped_freq (%lu) for CPUs [%*pbl]\n", + max_freq, capped_freq, cpumask_pr_args(cpus)); return; + } capacity = mult_frac(capped_freq, max_capacity, max_freq); ------------------------------>8--------------------------- Could you also dump for me the cpufreq and capacity sysfs content? $ grep . /sys/devices/system/cpu/cpu*/cpufreq/* $ grep . /sys/devices/system/cpu/cpu*/cpu_capacity Regards, Lukasz
Hi, On 11/5/21 11:26 AM, Lukasz Luba wrote: > Hi Steev, > > On 11/5/21 3:39 PM, Steev Klimaszewski wrote: >> Hi Lukasz, >> > > [snip] > >> I've been testing this patchset on the Lenovo Yoga C630, and today >> while compiling alacritty and running an apt-get full-upgrade, I >> found the following in dmesg output: > > Thank you for testing and sending feedback! > > Are you using a mainline kernel or you applied on some vendor production > kernel this patch set? I need to exclude a different code base > from the equation, especially to the arch_topology.c init code. > This is stabe 5.15.0 tree with ~72 (including your 6 patches on top (including the below as a patch). You can find it at https://github.com/steev/linux/commits/linux-5.15.y - the vast majority are just various fixups for sdm845/sdm850 for the Lenovo Yoga (or crypto since I'd like to see the crypto unit working). I did grep through my logs and it appears that this started after I moved from v2 to v3 - I'd tested v2 and it didn't show this. [snip] > [snip] > > That's interesting why we hit this. I should have added info about > those two values, which are compared. > > Could you make this change and try it again, please? > We would know the problematic values, which triggered this. > ---------------------8<----------------------------------- > diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c > index db18d79065fe..0d8db0927041 100644 > --- a/drivers/base/arch_topology.c > +++ b/drivers/base/arch_topology.c > @@ -185,8 +185,11 @@ void topology_update_thermal_pressure(const > struct cpumask *cpus, > /* Convert to MHz scale which is used in 'freq_factor' */ > capped_freq /= 1000; > > - if (WARN_ON(max_freq < capped_freq)) > + if (max_freq < capped_freq) { > + pr_warn("THERMAL_PRESSURE: max_freq (%lu) < > capped_freq (%lu) for CPUs [%*pbl]\n", > + max_freq, capped_freq, cpumask_pr_args(cpus)); > return; > + } > > capacity = mult_frac(capped_freq, max_capacity, max_freq); > > ------------------------------>8--------------------------- Applying this to the above kernel.. will test... > > Could you also dump for me the cpufreq and capacity sysfs content? > $ grep . /sys/devices/system/cpu/cpu*/cpufreq/* /sys/devices/system/cpu/cpu0/cpufreq/affected_cpus:0 1 2 3 /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq:300000 /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq:1766400 /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq:300000 /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu0/cpufreq/related_cpus:0 1 2 3 /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies:300000 403200 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 1324800 1420800 1516800 1612800 1689600 1766400 /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:300000 /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:1766400 /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq:300000 /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed:<unsupported> /sys/devices/system/cpu/cpu1/cpufreq/affected_cpus:0 1 2 3 /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq:300000 /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_max_freq:1766400 /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_min_freq:300000 /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu1/cpufreq/related_cpus:0 1 2 3 /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_frequencies:300000 403200 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 1324800 1420800 1516800 1612800 1689600 1766400 /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:300000 /sys/devices/system/cpu/cpu1/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:1766400 /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq:300000 /sys/devices/system/cpu/cpu1/cpufreq/scaling_setspeed:<unsupported> /sys/devices/system/cpu/cpu2/cpufreq/affected_cpus:0 1 2 3 /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_cur_freq:300000 /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_max_freq:1766400 /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_min_freq:300000 /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu2/cpufreq/related_cpus:0 1 2 3 /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_frequencies:300000 403200 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 1324800 1420800 1516800 1612800 1689600 1766400 /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:300000 /sys/devices/system/cpu/cpu2/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq:1766400 /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq:300000 /sys/devices/system/cpu/cpu2/cpufreq/scaling_setspeed:<unsupported> /sys/devices/system/cpu/cpu3/cpufreq/affected_cpus:0 1 2 3 /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_cur_freq:300000 /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_max_freq:1766400 /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_min_freq:300000 /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu3/cpufreq/related_cpus:0 1 2 3 /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_frequencies:300000 403200 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 1324800 1420800 1516800 1612800 1689600 1766400 /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:300000 /sys/devices/system/cpu/cpu3/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq:1766400 /sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq:300000 /sys/devices/system/cpu/cpu3/cpufreq/scaling_setspeed:<unsupported> /sys/devices/system/cpu/cpu4/cpufreq/affected_cpus:4 5 6 7 /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_cur_freq:1920000 /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_max_freq:2956800 /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_min_freq:825600 /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu4/cpufreq/related_cpus:4 5 6 7 /sys/devices/system/cpu/cpu4/cpufreq/scaling_available_frequencies:825600 902400 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 2476800 2553600 2649600 2745600 2841600 /sys/devices/system/cpu/cpu4/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu4/cpufreq/scaling_boost_frequencies:2956800 /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:1920000 /sys/devices/system/cpu/cpu4/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq:2841600 /sys/devices/system/cpu/cpu4/cpufreq/scaling_min_freq:825600 /sys/devices/system/cpu/cpu4/cpufreq/scaling_setspeed:<unsupported> /sys/devices/system/cpu/cpu5/cpufreq/affected_cpus:4 5 6 7 /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_cur_freq:1996800 /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_max_freq:2956800 /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_min_freq:825600 /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu5/cpufreq/related_cpus:4 5 6 7 /sys/devices/system/cpu/cpu5/cpufreq/scaling_available_frequencies:825600 902400 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 2476800 2553600 2649600 2745600 2841600 /sys/devices/system/cpu/cpu5/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu5/cpufreq/scaling_boost_frequencies:2956800 /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:1996800 /sys/devices/system/cpu/cpu5/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq:2841600 /sys/devices/system/cpu/cpu5/cpufreq/scaling_min_freq:825600 /sys/devices/system/cpu/cpu5/cpufreq/scaling_setspeed:<unsupported> /sys/devices/system/cpu/cpu6/cpufreq/affected_cpus:4 5 6 7 /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_cur_freq:1996800 /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_max_freq:2956800 /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_min_freq:825600 /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu6/cpufreq/related_cpus:4 5 6 7 /sys/devices/system/cpu/cpu6/cpufreq/scaling_available_frequencies:825600 902400 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 2476800 2553600 2649600 2745600 2841600 /sys/devices/system/cpu/cpu6/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu6/cpufreq/scaling_boost_frequencies:2956800 /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:1996800 /sys/devices/system/cpu/cpu6/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq:2841600 /sys/devices/system/cpu/cpu6/cpufreq/scaling_min_freq:825600 /sys/devices/system/cpu/cpu6/cpufreq/scaling_setspeed:<unsupported> /sys/devices/system/cpu/cpu7/cpufreq/affected_cpus:4 5 6 7 /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_cur_freq:1996800 /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_max_freq:2956800 /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_min_freq:825600 /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_transition_latency:0 /sys/devices/system/cpu/cpu7/cpufreq/related_cpus:4 5 6 7 /sys/devices/system/cpu/cpu7/cpufreq/scaling_available_frequencies:825600 902400 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 2476800 2553600 2649600 2745600 2841600 /sys/devices/system/cpu/cpu7/cpufreq/scaling_available_governors:ondemand conservative powersave userspace performance schedutil /sys/devices/system/cpu/cpu7/cpufreq/scaling_boost_frequencies:2956800 /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:1996800 /sys/devices/system/cpu/cpu7/cpufreq/scaling_driver:qcom-cpufreq-hw /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor:schedutil /sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq:2841600 /sys/devices/system/cpu/cpu7/cpufreq/scaling_min_freq:825600 /sys/devices/system/cpu/cpu7/cpufreq/scaling_setspeed:<unsupported> > $ grep . /sys/devices/system/cpu/cpu*/cpu_capacity /sys/devices/system/cpu/cpu0/cpu_capacity:377 /sys/devices/system/cpu/cpu1/cpu_capacity:377 /sys/devices/system/cpu/cpu2/cpu_capacity:377 /sys/devices/system/cpu/cpu3/cpu_capacity:377 /sys/devices/system/cpu/cpu4/cpu_capacity:1024 /sys/devices/system/cpu/cpu5/cpu_capacity:1024 /sys/devices/system/cpu/cpu6/cpu_capacity:1024 /sys/devices/system/cpu/cpu7/cpu_capacity:1024 In taking a look at cpufreq-info, one thing I noticed is that even though I have 1 in /sys/devices/system/cpu/cpufreq/boost, I am *never* hitting the 2.96GHz now cpufreq stats: 826 MHz:59.14%, 902 MHz:0.15%, 979 MHz:0.18%, 1.06 GHz:0.11%, 1.21 GHz:0.49%, 1.29 GHz:0.26%, 1.36 GHz:0.12%, 1.46 GHz:0.23%, 1.54 GHz:0.10%, 1.61 GHz:0.14%, 1.69 GHz:0.09%, 1.77 GHz:0.28%, 1.84 GHz:0.64%, 1.92 GHz:0.23%, 2.00 GHz:0.05%, 2.09 GHz:0.05%, 2.17 GHz:0.03%, 2.25 GHz:0.03%, 2.32 GHz:0.03%, 2.40 GHz:0.03%, 2.48 GHz:0.02%, 2.55 GHz:0.02%, 2.65 GHz:0.03%, 2.75 GHz:0.03%, 2.84 GHz:37.53%, 2.96 GHz:0.00% (20854) Aaaand it looks like that is part of the deal - with your patch from above applied, we get: [ 22.487268] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for CPUs [4-7] [ 22.487313] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for CPUs [4-7] [ 22.508642] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for CPUs [4-7] [ 22.552273] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for CPUs [4-7] So, we're not able to hit boost frequencies with this applied? Thank you for the fast response! -- steev
On 11/5/21 1:33 PM, Steev Klimaszewski wrote: > Hi, > > On 11/5/21 11:26 AM, Lukasz Luba wrote: >> Hi Steev, >> >> On 11/5/21 3:39 PM, Steev Klimaszewski wrote: >>> Hi Lukasz, >>> >> >> [snip] >> >>> I've been testing this patchset on the Lenovo Yoga C630, and today >>> while compiling alacritty and running an apt-get full-upgrade, I >>> found the following in dmesg output: >> >> Thank you for testing and sending feedback! >> >> Are you using a mainline kernel or you applied on some vendor production >> kernel this patch set? I need to exclude a different code base >> from the equation, especially to the arch_topology.c init code. >> > This is stabe 5.15.0 tree with ~72 (including your 6 patches on top > (including the below as a patch). You can find it at > https://github.com/steev/linux/commits/linux-5.15.y - the vast majority > are just various fixups for sdm845/sdm850 for the Lenovo Yoga (or crypto > since I'd like to see the crypto unit working). > > I did grep through my logs and it appears that this started after I > moved from v2 to v3 - I'd tested v2 and it didn't show this. > > [snip] > >> [snip] >> >> That's interesting why we hit this. I should have added info about >> those two values, which are compared. >> >> Could you make this change and try it again, please? >> We would know the problematic values, which triggered this. >> ---------------------8<----------------------------------- >> diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c >> index db18d79065fe..0d8db0927041 100644 >> --- a/drivers/base/arch_topology.c >> +++ b/drivers/base/arch_topology.c >> @@ -185,8 +185,11 @@ void topology_update_thermal_pressure(const >> struct cpumask *cpus, >> /* Convert to MHz scale which is used in 'freq_factor' */ >> capped_freq /= 1000; >> >> - if (WARN_ON(max_freq < capped_freq)) >> + if (max_freq < capped_freq) { >> + pr_warn("THERMAL_PRESSURE: max_freq (%lu) < >> capped_freq (%lu) for CPUs [%*pbl]\n", >> + max_freq, capped_freq, cpumask_pr_args(cpus)); >> return; >> + } >> >> capacity = mult_frac(capped_freq, max_capacity, max_freq); >> >> ------------------------------>8--------------------------- > > Applying this to the above kernel.. will test... > > >> >> Could you also dump for me the cpufreq and capacity sysfs content? >> $ grep . /sys/devices/system/cpu/cpu*/cpufreq/* > > > /sys/devices/system/cpu/cpu0/cpufreq/affected_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq:300000 > /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq:1766400 > /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq:300000 > /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu0/cpufreq/related_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies:300000 403200 > 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 > 1324800 1420800 1516800 1612800 1689600 1766400 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:300000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:1766400 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq:300000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed:<unsupported> > /sys/devices/system/cpu/cpu1/cpufreq/affected_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq:300000 > /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_max_freq:1766400 > /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_min_freq:300000 > /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu1/cpufreq/related_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_frequencies:300000 403200 > 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 > 1324800 1420800 1516800 1612800 1689600 1766400 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:300000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:1766400 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq:300000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_setspeed:<unsupported> > /sys/devices/system/cpu/cpu2/cpufreq/affected_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_cur_freq:300000 > /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_max_freq:1766400 > /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_min_freq:300000 > /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu2/cpufreq/related_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_frequencies:300000 403200 > 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 > 1324800 1420800 1516800 1612800 1689600 1766400 > /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:300000 > /sys/devices/system/cpu/cpu2/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq:1766400 > /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq:300000 > /sys/devices/system/cpu/cpu2/cpufreq/scaling_setspeed:<unsupported> > /sys/devices/system/cpu/cpu3/cpufreq/affected_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_cur_freq:300000 > /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_max_freq:1766400 > /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_min_freq:300000 > /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu3/cpufreq/related_cpus:0 1 2 3 > /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_frequencies:300000 403200 > 480000 576000 652800 748800 825600 902400 979200 1056000 1132800 1228800 > 1324800 1420800 1516800 1612800 1689600 1766400 > /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:300000 > /sys/devices/system/cpu/cpu3/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq:1766400 > /sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq:300000 > /sys/devices/system/cpu/cpu3/cpufreq/scaling_setspeed:<unsupported> > /sys/devices/system/cpu/cpu4/cpufreq/affected_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_cur_freq:1920000 > /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_max_freq:2956800 > /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_min_freq:825600 > /sys/devices/system/cpu/cpu4/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu4/cpufreq/related_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu4/cpufreq/scaling_available_frequencies:825600 902400 > 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 > 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 > 2476800 2553600 2649600 2745600 2841600 > /sys/devices/system/cpu/cpu4/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu4/cpufreq/scaling_boost_frequencies:2956800 > /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq:1920000 > /sys/devices/system/cpu/cpu4/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq:2841600 > /sys/devices/system/cpu/cpu4/cpufreq/scaling_min_freq:825600 > /sys/devices/system/cpu/cpu4/cpufreq/scaling_setspeed:<unsupported> > /sys/devices/system/cpu/cpu5/cpufreq/affected_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_cur_freq:1996800 > /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_max_freq:2956800 > /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_min_freq:825600 > /sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu5/cpufreq/related_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu5/cpufreq/scaling_available_frequencies:825600 902400 > 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 > 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 > 2476800 2553600 2649600 2745600 2841600 > /sys/devices/system/cpu/cpu5/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu5/cpufreq/scaling_boost_frequencies:2956800 > /sys/devices/system/cpu/cpu5/cpufreq/scaling_cur_freq:1996800 > /sys/devices/system/cpu/cpu5/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq:2841600 > /sys/devices/system/cpu/cpu5/cpufreq/scaling_min_freq:825600 > /sys/devices/system/cpu/cpu5/cpufreq/scaling_setspeed:<unsupported> > /sys/devices/system/cpu/cpu6/cpufreq/affected_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_cur_freq:1996800 > /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_max_freq:2956800 > /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_min_freq:825600 > /sys/devices/system/cpu/cpu6/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu6/cpufreq/related_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu6/cpufreq/scaling_available_frequencies:825600 902400 > 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 > 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 > 2476800 2553600 2649600 2745600 2841600 > /sys/devices/system/cpu/cpu6/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu6/cpufreq/scaling_boost_frequencies:2956800 > /sys/devices/system/cpu/cpu6/cpufreq/scaling_cur_freq:1996800 > /sys/devices/system/cpu/cpu6/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq:2841600 > /sys/devices/system/cpu/cpu6/cpufreq/scaling_min_freq:825600 > /sys/devices/system/cpu/cpu6/cpufreq/scaling_setspeed:<unsupported> > /sys/devices/system/cpu/cpu7/cpufreq/affected_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_cur_freq:1996800 > /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_max_freq:2956800 > /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_min_freq:825600 > /sys/devices/system/cpu/cpu7/cpufreq/cpuinfo_transition_latency:0 > /sys/devices/system/cpu/cpu7/cpufreq/related_cpus:4 5 6 7 > /sys/devices/system/cpu/cpu7/cpufreq/scaling_available_frequencies:825600 902400 > 979200 1056000 1209600 1286400 1363200 1459200 1536000 1612800 1689600 > 1766400 1843200 1920000 1996800 2092800 2169600 2246400 2323200 2400000 > 2476800 2553600 2649600 2745600 2841600 > /sys/devices/system/cpu/cpu7/cpufreq/scaling_available_governors:ondemand conservative > powersave userspace performance schedutil > /sys/devices/system/cpu/cpu7/cpufreq/scaling_boost_frequencies:2956800 > /sys/devices/system/cpu/cpu7/cpufreq/scaling_cur_freq:1996800 > /sys/devices/system/cpu/cpu7/cpufreq/scaling_driver:qcom-cpufreq-hw > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor:schedutil > /sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq:2841600 > /sys/devices/system/cpu/cpu7/cpufreq/scaling_min_freq:825600 > /sys/devices/system/cpu/cpu7/cpufreq/scaling_setspeed:<unsupported> > > >> $ grep . /sys/devices/system/cpu/cpu*/cpu_capacity > > /sys/devices/system/cpu/cpu0/cpu_capacity:377 > /sys/devices/system/cpu/cpu1/cpu_capacity:377 > /sys/devices/system/cpu/cpu2/cpu_capacity:377 > /sys/devices/system/cpu/cpu3/cpu_capacity:377 > /sys/devices/system/cpu/cpu4/cpu_capacity:1024 > /sys/devices/system/cpu/cpu5/cpu_capacity:1024 > /sys/devices/system/cpu/cpu6/cpu_capacity:1024 > /sys/devices/system/cpu/cpu7/cpu_capacity:1024 > > > In taking a look at cpufreq-info, one thing I noticed is that even > though I have 1 in /sys/devices/system/cpu/cpufreq/boost, I am *never* > hitting the 2.96GHz now > > cpufreq stats: 826 MHz:59.14%, 902 MHz:0.15%, 979 MHz:0.18%, 1.06 > GHz:0.11%, 1.21 GHz:0.49%, 1.29 GHz:0.26%, 1.36 GHz:0.12%, 1.46 > GHz:0.23%, 1.54 GHz:0.10%, 1.61 GHz:0.14%, 1.69 GHz:0.09%, 1.77 > GHz:0.28%, 1.84 GHz:0.64%, 1.92 GHz:0.23%, 2.00 GHz:0.05%, 2.09 > GHz:0.05%, 2.17 GHz:0.03%, 2.25 GHz:0.03%, 2.32 GHz:0.03%, 2.40 > GHz:0.03%, 2.48 GHz:0.02%, 2.55 GHz:0.02%, 2.65 GHz:0.03%, 2.75 > GHz:0.03%, 2.84 GHz:37.53%, 2.96 GHz:0.00% (20854) > > Aaaand it looks like that is part of the deal - with your patch from > above applied, we get: > > [ 22.487268] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for > CPUs [4-7] > [ 22.487313] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for > CPUs [4-7] > [ 22.508642] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for > CPUs [4-7] > [ 22.552273] THERMAL_PRESSURE: max_freq(2841) < capped_freq(2956) for > CPUs [4-7] > > So, we're not able to hit boost frequencies with this applied? Hi Steve, Does your system have enough load to hit the boost frequencies ? I don't think this patch should affect hitting boost frequencies as there is no error being returned from topology_update_thermal_pressure. The warning you are getting is because you have boost frequency enabled and IIUC lmh enabled and thermal pressure framework bails out due to boost_frequency being greater than what is available in per_cpu freq_factor. This is because we do not recalculate freq_factor every time boost is enabled / disabled. IIRC there were some discussions around rebuilding scheduler domains and capacity with user space changes to max frequency but it has never proceeded much. Till that point, I think the right way, is to check whether the new capcity exceeds the max_capacity of the cpu and if yes use max_capacity in lieu of new_capacity to calculate thermal pressure. > > Thank you for the fast response! > > -- steev >
On 11/5/21 2:18 PM, Thara Gopinath wrote: > > > On 11/5/21 1:33 PM, Steev Klimaszewski wrote: >> Hi, >> >> On 11/5/21 11:26 AM, Lukasz Luba wrote: >>> Hi Steev, >>> >>> On 11/5/21 3:39 PM, Steev Klimaszewski wrote: >>>> Hi Lukasz, >>>> >>> [snip] > Hi Steve, > > Does your system have enough load to hit the boost frequencies ? I > don't think this patch should affect hitting boost frequencies as > there is no error being returned from topology_update_thermal_pressure. > > The warning you are getting is because you have boost frequency > enabled and IIUC lmh enabled and thermal pressure framework bails out > due to boost_frequency being greater than what is available in per_cpu > freq_factor. This is because we do not recalculate freq_factor every > time boost is enabled / disabled. IIRC there were some discussions > around rebuilding scheduler domains and capacity with user space > changes to max frequency but it has never proceeded much. Till that > point, I think the right way, is to check whether the new capcity > exceeds the max_capacity of the cpu and if yes use max_capacity in > lieu of new_capacity to calculate thermal pressure. > Hi Thara, I should definitely be able to push it to 2.96GHz, however I'm simply not getting it at all with these patches applied. So, I'm currently compiling multiple applications - alacritty (https://github.com/alacritty/alacritty), and zellij (https://github.com/zellij-org/zellij), as well as running pixz on a 5.1GB file to compress it, and throwing in cpuburn-a53 (https://github.com/ssvb/cpuburn-arm) and I'm simply not getting 2.96GHz at all. Ever. I don't normally try to push it that high, but I wanted to see if we could ever hit it (the system was also never going above 86C) analyzing CPU 4: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:54.84%, 902 MHz:0.02%, 979 MHz:0.02%, 1.06 GHz:0.02%, 1.21 GHz:0.08%, 1.29 GHz:0.07%, 1.36 GHz:0.09%, 1.46 GHz:0.04%, 1.54 GHz:0.02%, 1.61 GHz:0.02%, 1.69 GHz:0.02%, 1.77 GHz:0.13%, 1.84 GHz:0.04%, 1.92 GHz:0.04%, 2.00 GHz:0.02%, 2.09 GHz:0.03%, 2.17 GHz:0.02%, 2.25 GHz:0.02%, 2.32 GHz:0.01%, 2.40 GHz:0.02%, 2.48 GHz:0.02%, 2.55 GHz:0.02%, 2.65 GHz:0.02%, 2.75 GHz:0.02%, 2.84 GHz:44.38%, 2.96 GHz:0.00% (8066) analyzing CPU 5: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:54.84%, 902 MHz:0.02%, 979 MHz:0.02%, 1.06 GHz:0.02%, 1.21 GHz:0.08%, 1.29 GHz:0.07%, 1.36 GHz:0.09%, 1.46 GHz:0.04%, 1.54 GHz:0.02%, 1.61 GHz:0.02%, 1.69 GHz:0.02%, 1.77 GHz:0.13%, 1.84 GHz:0.04%, 1.92 GHz:0.04%, 2.00 GHz:0.02%, 2.09 GHz:0.03%, 2.17 GHz:0.02%, 2.25 GHz:0.02%, 2.32 GHz:0.01%, 2.40 GHz:0.02%, 2.48 GHz:0.02%, 2.55 GHz:0.02%, 2.65 GHz:0.02%, 2.75 GHz:0.02%, 2.84 GHz:44.38%, 2.96 GHz:0.00% (8066) analyzing CPU 6: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:54.84%, 902 MHz:0.02%, 979 MHz:0.02%, 1.06 GHz:0.02%, 1.21 GHz:0.08%, 1.29 GHz:0.07%, 1.36 GHz:0.09%, 1.46 GHz:0.04%, 1.54 GHz:0.02%, 1.61 GHz:0.02%, 1.69 GHz:0.02%, 1.77 GHz:0.13%, 1.84 GHz:0.04%, 1.92 GHz:0.04%, 2.00 GHz:0.02%, 2.09 GHz:0.03%, 2.17 GHz:0.02%, 2.25 GHz:0.02%, 2.32 GHz:0.01%, 2.40 GHz:0.02%, 2.48 GHz:0.02%, 2.55 GHz:0.02%, 2.65 GHz:0.02%, 2.75 GHz:0.02%, 2.84 GHz:44.38%, 2.96 GHz:0.00% (8066) analyzing CPU 7: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:54.84%, 902 MHz:0.02%, 979 MHz:0.02%, 1.06 GHz:0.02%, 1.21 GHz:0.08%, 1.29 GHz:0.07%, 1.36 GHz:0.09%, 1.46 GHz:0.04%, 1.54 GHz:0.02%, 1.61 GHz:0.02%, 1.69 GHz:0.02%, 1.77 GHz:0.13%, 1.84 GHz:0.04%, 1.92 GHz:0.04%, 2.00 GHz:0.02%, 2.09 GHz:0.03%, 2.17 GHz:0.02%, 2.25 GHz:0.02%, 2.32 GHz:0.01%, 2.40 GHz:0.02%, 2.48 GHz:0.02%, 2.55 GHz:0.02%, 2.65 GHz:0.02%, 2.75 GHz:0.02%, 2.84 GHz:44.38%, 2.96 GHz:0.00% (8066) After removing this patchset, and rebooting and just compiling zellij: analyzing CPU 4: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:16.01%, 902 MHz:0.08%, 979 MHz:0.05%, 1.06 GHz:0.06%, 1.21 GHz:0.37%, 1.29 GHz:0.17%, 1.36 GHz:0.15%, 1.46 GHz:0.20%, 1.54 GHz:0.18%, 1.61 GHz:0.21%, 1.69 GHz:0.17%, 1.77 GHz:0.22%, 1.84 GHz:0.32%, 1.92 GHz:0.37%, 2.00 GHz:0.22%, 2.09 GHz:0.20%, 2.17 GHz:0.20%, 2.25 GHz:0.19%, 2.32 GHz:0.19%, 2.40 GHz:0.21%, 2.48 GHz:0.18%, 2.55 GHz:0.18%, 2.65 GHz:0.21%, 2.75 GHz:0.16%, 2.84 GHz:79.49%, 2.96 GHz:0.03% (5315) analyzing CPU 5: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:16.01%, 902 MHz:0.08%, 979 MHz:0.05%, 1.06 GHz:0.06%, 1.21 GHz:0.37%, 1.29 GHz:0.17%, 1.36 GHz:0.15%, 1.46 GHz:0.20%, 1.54 GHz:0.18%, 1.61 GHz:0.21%, 1.69 GHz:0.17%, 1.77 GHz:0.22%, 1.84 GHz:0.32%, 1.92 GHz:0.37%, 2.00 GHz:0.22%, 2.09 GHz:0.20%, 2.17 GHz:0.20%, 2.25 GHz:0.19%, 2.32 GHz:0.19%, 2.40 GHz:0.21%, 2.48 GHz:0.18%, 2.55 GHz:0.18%, 2.65 GHz:0.21%, 2.75 GHz:0.16%, 2.84 GHz:79.49%, 2.96 GHz:0.03% (5315) analyzing CPU 6: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:16.01%, 902 MHz:0.08%, 979 MHz:0.05%, 1.06 GHz:0.06%, 1.21 GHz:0.37%, 1.29 GHz:0.17%, 1.36 GHz:0.15%, 1.46 GHz:0.20%, 1.54 GHz:0.18%, 1.61 GHz:0.21%, 1.69 GHz:0.17%, 1.77 GHz:0.22%, 1.84 GHz:0.32%, 1.92 GHz:0.37%, 2.00 GHz:0.22%, 2.09 GHz:0.20%, 2.17 GHz:0.20%, 2.25 GHz:0.19%, 2.32 GHz:0.19%, 2.40 GHz:0.21%, 2.48 GHz:0.18%, 2.55 GHz:0.18%, 2.65 GHz:0.21%, 2.75 GHz:0.16%, 2.84 GHz:79.49%, 2.96 GHz:0.03% (5315) analyzing CPU 7: driver: qcom-cpufreq-hw CPUs which run at the same hardware frequency: 4 5 6 7 CPUs which need to have their frequency coordinated by software: 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 826 MHz - 2.96 GHz available frequency steps: 826 MHz, 902 MHz, 979 MHz, 1.06 GHz, 1.21 GHz, 1.29 GHz, 1.36 GHz, 1.46 GHz, 1.54 GHz, 1.61 GHz, 1.69 GHz, 1.77 GHz, 1.84 GHz, 1.92 GHz, 2.00 GHz, 2.09 GHz, 2.17 GHz, 2.25 GHz, 2.32 GHz, 2.40 GHz, 2.48 GHz, 2.55 GHz, 2.65 GHz, 2.75 GHz, 2.84 GHz available cpufreq governors: ondemand, conservative, powersave, userspace, performance, schedutil current policy: frequency should be within 826 MHz and 2.84 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 2.84 GHz. cpufreq stats: 826 MHz:16.01%, 902 MHz:0.08%, 979 MHz:0.05%, 1.06 GHz:0.06%, 1.21 GHz:0.37%, 1.29 GHz:0.17%, 1.36 GHz:0.15%, 1.46 GHz:0.20%, 1.54 GHz:0.18%, 1.61 GHz:0.21%, 1.69 GHz:0.17%, 1.77 GHz:0.22%, 1.84 GHz:0.32%, 1.92 GHz:0.37%, 2.00 GHz:0.22%, 2.09 GHz:0.20%, 2.17 GHz:0.20%, 2.25 GHz:0.19%, 2.32 GHz:0.19%, 2.40 GHz:0.21%, 2.48 GHz:0.18%, 2.55 GHz:0.18%, 2.65 GHz:0.21%, 2.75 GHz:0.16%, 2.84 GHz:79.49%, 2.96 GHz:0.03% (5315) >> >> Thank you for the fast response! >> >> -- steev >> >
On 11/5/21 3:51 PM, Steev Klimaszewski wrote: > > On 11/5/21 2:18 PM, Thara Gopinath wrote: >> >> >> On 11/5/21 1:33 PM, Steev Klimaszewski wrote: >>> Hi, >>> >>> On 11/5/21 11:26 AM, Lukasz Luba wrote: >>>> Hi Steev, >>>> >>>> On 11/5/21 3:39 PM, Steev Klimaszewski wrote: >>>>> Hi Lukasz, >>>>> >>>> [snip] >> Hi Steve, >> >> Does your system have enough load to hit the boost frequencies ? I >> don't think this patch should affect hitting boost frequencies as >> there is no error being returned from topology_update_thermal_pressure. >> >> The warning you are getting is because you have boost frequency >> enabled and IIUC lmh enabled and thermal pressure framework bails out >> due to boost_frequency being greater than what is available in per_cpu >> freq_factor. This is because we do not recalculate freq_factor every >> time boost is enabled / disabled. IIRC there were some discussions >> around rebuilding scheduler domains and capacity with user space >> changes to max frequency but it has never proceeded much. Till that >> point, I think the right way, is to check whether the new capcity >> exceeds the max_capacity of the cpu and if yes use max_capacity in >> lieu of new_capacity to calculate thermal pressure. >> > Hi Thara, > > I should definitely be able to push it to 2.96GHz, however I'm simply > not getting it at all with these patches applied. > > So, I'm currently compiling multiple applications - alacritty > (https://github.com/alacritty/alacritty), and zellij > (https://github.com/zellij-org/zellij), as well as running pixz on a > 5.1GB file to compress it, and throwing in cpuburn-a53 > (https://github.com/ssvb/cpuburn-arm) and I'm simply not getting 2.96GHz > at all. Ever. I don't normally try to push it that high, but I wanted > to see if we could ever hit it (the system was also never going above 86C) Hi, So IIUC the below logs correctly, you are never hitting boost frequency (with or without this patch series). Is that correct ? w.r.t temperature , how are you measuring it? Do you have LMh enabled or are you using tsens to mitigate cpu temperature ?
> [snip] > Hi, > > So IIUC the below logs correctly, you are never hitting boost > frequency (with or without this patch series). Is that correct ? > > w.r.t temperature , how are you measuring it? Do you have LMh enabled > or are you using tsens to mitigate cpu temperature ? Hi, I was wrong - it does indeed go boost with the patchset applied, it's just that it doesn't boost up to 2.96GHz very often at all. As noted by the 0.03% when i ran it while compiling zellij; I reapplied the patches (and the 6th patch from Lukasz's email) and after boot, 2.96GHz was showing at 0.39%. Most tools that read the cpu frequency don't really seem to be well suited for big.LITTLE, and seem to throw an average of the speed, so cpufreq-info was the best I have. We're apparently supposed to be using cpupower these days, but it doesn't seem to know anything about arm64 devices. Temperature wise, I'm just getting from the sensors, and I am using LMh. Now, I have to admit, while I've thrown a patch here or there, I'm not exactly a kernel developer, just enough knowledge to be somewhat dangerous and know how to backport things. In my mind, and my line of thinking, I would expect with boost enabled, that the cpu would boost up to that as often as possible, not require a specific workload to actually hit it. But then again, I would expect multiple compilation jobs to be one of the workloads that would? So I think, the part about never hitting 2.96GHz can be dismissed, and was simply my lack of knowledge about the cpufreq-info tool's averages. It does seem however to rarely ever hit 2.96GHz and I would actually expect it to hit it far more often.
On 11/5/21 10:46 PM, Steev Klimaszewski wrote: > >> [snip] >> Hi, >> >> So IIUC the below logs correctly, you are never hitting boost >> frequency (with or without this patch series). Is that correct ? >> >> w.r.t temperature , how are you measuring it? Do you have LMh enabled >> or are you using tsens to mitigate cpu temperature ? > > > Hi, > > I was wrong - it does indeed go boost with the patchset applied, it's > just that it doesn't boost up to 2.96GHz very often at all. As noted by > the 0.03% when i ran it while compiling zellij; I reapplied the patches > (and the 6th patch from Lukasz's email) and after boot, 2.96GHz was > showing at 0.39%. > > Most tools that read the cpu frequency don't really seem to be well > suited for big.LITTLE, and seem to throw an average of the speed, so > cpufreq-info was the best I have. We're apparently supposed to be using > cpupower these days, but it doesn't seem to know anything about arm64 > devices. > > Temperature wise, I'm just getting from the sensors, and I am using LMh. > > Now, I have to admit, while I've thrown a patch here or there, I'm not > exactly a kernel developer, just enough knowledge to be somewhat > dangerous and know how to backport things. In my mind, and my line of > thinking, I would expect with boost enabled, that the cpu would boost up > to that as often as possible, not require a specific workload to > actually hit it. But then again, I would expect multiple compilation > jobs to be one of the workloads that would? > > So I think, the part about never hitting 2.96GHz can be dismissed, and > was simply my lack of knowledge about the cpufreq-info tool's averages. > It does seem however to rarely ever hit 2.96GHz and I would actually > expect it to hit it far more often. > Thank you for the logs and re-testing it with the debug changes. That's valuable information. I'll work on it today. Regarding the 'boost' frequency, as you observed, it's hard to measure it properly. Normally when you use the frequency governor 'schedutil' (which is your case), you need a high utilization workload to request highest frequency. The task scheduler does this calculation and then asks for the frequency. I'll spend more time trying to understand how this Qcom driver and HW handles it. The patch set itself should not block it. I'll get back to you soon. Regards, Lukasz
On 11/5/21 6:46 PM, Steev Klimaszewski wrote: > >> [snip] >> Hi, >> >> So IIUC the below logs correctly, you are never hitting boost >> frequency (with or without this patch series). Is that correct ? >> >> w.r.t temperature , how are you measuring it? Do you have LMh enabled >> or are you using tsens to mitigate cpu temperature ? > > > Hi, > > I was wrong - it does indeed go boost with the patchset applied, it's > just that it doesn't boost up to 2.96GHz very often at all. As noted by > the 0.03% when i ran it while compiling zellij; I reapplied the patches > (and the 6th patch from Lukasz's email) and after boot, 2.96GHz was > showing at 0.39%. > > Most tools that read the cpu frequency don't really seem to be well > suited for big.LITTLE, and seem to throw an average of the speed, so > cpufreq-info was the best I have. We're apparently supposed to be using > cpupower these days, but it doesn't seem to know anything about arm64 > devices. > > Temperature wise, I'm just getting from the sensors, and I am using LMh. > > Now, I have to admit, while I've thrown a patch here or there, I'm not > exactly a kernel developer, just enough knowledge to be somewhat > dangerous and know how to backport things. In my mind, and my line of > thinking, I would expect with boost enabled, that the cpu would boost up > to that as often as possible, not require a specific workload to > actually hit it. But then again, I would expect multiple compilation > jobs to be one of the workloads that would? Hi Steev, So this depends on the cpufreq governor you are using. By-default arm systems have sched-util governor enabled. This means you will scale up to boost depending on cpu load and not always. If you want to ensure you are always hitting boost frequency, you should enable performance governor for cpufreq and try. Also since the defconfig has by default CPU_FREQ_STAT enabled, you should be able to get statistics out of cpufreq to see the time spent by a cpu in each frequency. I think cpufreq-info -s should give you this info. If not, you can explicitly get it for each cpu from cat /sys/devices/system/cpu/cpu<X>/cpufreq/stats/time_in_state Regarding temperature, if you have applied all the patches in the sdm845 LMh series and have LMh enabled, cpu throttling starts around 95 degree C. > > So I think, the part about never hitting 2.96GHz can be dismissed, and > was simply my lack of knowledge about the cpufreq-info tool's averages. > It does seem however to rarely ever hit 2.96GHz and I would actually > expect it to hit it far more often. >
> Hi Steev, > > So this depends on the cpufreq governor you are using. By-default arm > systems have sched-util governor enabled. This means you will scale up > to boost depending on cpu load and not always. If you want to ensure > you are always hitting boost frequency, you should enable performance > governor for cpufreq and try. > > Also since the defconfig has by default CPU_FREQ_STAT enabled, you > should be able to get statistics out of cpufreq to see the time spent > by a cpu in each frequency. I think cpufreq-info -s should give you > this info. If not, you can explicitly get it for each cpu from > > cat /sys/devices/system/cpu/cpu<X>/cpufreq/stats/time_in_state > > Regarding temperature, if you have applied all the patches in the > sdm845 LMh series and have LMh enabled, cpu throttling starts around > 95 degree C. > Hi Thara, Indeed, I ended up finding the time_in_state when I was doing more digging after my last mail. I do have the sdm845 LMh series and LMh enabled, however I don't think I've ever seen my system go above 90C here. So a quick look, and... we are simply almost never getting the 2.95GHz at all, regardless of workload. I saw Lukasz response as well about the math possibly being wrong, but I haven't had a chance. Regarding the time in state - I went with policy4 instead of per cpu (for brevity sake) and it's here: c630:~$ cat /sys/devices/system/cpu/cpufreq/policy4/stats/time_in_state 825600 225037 902400 92 979200 205 1056000 96 1209600 902 1286400 386 1363200 396 1459200 217 1536000 101 1612800 75 1689600 95 1766400 130 1843200 255 1920000 318 1996800 92 2092800 87 2169600 66 2246400 60 2323200 58 2400000 54 2476800 47 2553600 50 2649600 69 2745600 58 2841600 54619 2956800 5 So we spend *very* little time in 2.96GHz and this is after almost 14 hours of uptime on the C630. By comparison, on a Pinebook Pro where I've added in 2GHz as a boost frequency : pinebook-pro:~$ cat /sys/devices/system/cpu/cpufreq/policy4/stats/time_in_state 408000 16084466 600000 27212 816000 32487 1008000 11331 1200000 13268 1416000 75078 1608000 18392 1800000 207266 2016000 648612 With the Pinebook Pro, which doesn't even come close to getting to 95C, we spend a lot more time in 2GHz. -- steev
On 11/8/21 10:22 AM, Steev Klimaszewski wrote: > >> Hi Steev, >> >> So this depends on the cpufreq governor you are using. By-default arm >> systems have sched-util governor enabled. This means you will scale up >> to boost depending on cpu load and not always. If you want to ensure >> you are always hitting boost frequency, you should enable performance >> governor for cpufreq and try. >> >> Also since the defconfig has by default CPU_FREQ_STAT enabled, you >> should be able to get statistics out of cpufreq to see the time spent >> by a cpu in each frequency. I think cpufreq-info -s should give you >> this info. If not, you can explicitly get it for each cpu from >> >> cat /sys/devices/system/cpu/cpu<X>/cpufreq/stats/time_in_state >> >> Regarding temperature, if you have applied all the patches in the >> sdm845 LMh series and have LMh enabled, cpu throttling starts around >> 95 degree C. >> > Hi Thara, > > Indeed, I ended up finding the time_in_state when I was doing more > digging after my last mail. I do have the sdm845 LMh series and LMh > enabled, however I don't think I've ever seen my system go above 90C here. > > So a quick look, and... we are simply almost never getting the 2.95GHz > at all, regardless of workload. I saw Lukasz response as well about the > math possibly being wrong, but I haven't had a chance. > > Regarding the time in state - I went with policy4 instead of per cpu > (for brevity sake) and it's here: > > c630:~$ cat /sys/devices/system/cpu/cpufreq/policy4/stats/time_in_state > 825600 225037 > 902400 92 > 979200 205 > 1056000 96 > 1209600 902 > 1286400 386 > 1363200 396 > 1459200 217 > 1536000 101 > 1612800 75 > 1689600 95 > 1766400 130 > 1843200 255 > 1920000 318 > 1996800 92 > 2092800 87 > 2169600 66 > 2246400 60 > 2323200 58 > 2400000 54 > 2476800 47 > 2553600 50 > 2649600 69 > 2745600 58 > 2841600 54619 > 2956800 5 > > So we spend *very* little time in 2.96GHz and this is after almost 14 > hours of uptime on the C630. By comparison, on a Pinebook Pro where > I've added in 2GHz as a boost frequency : Hi Steev, IIUC, PineBook Pro has Rockchip RK3399 which has 2 Cortex A-72 and 4 Cortex A-52 where as C630 has Qualcomm sdm845 which has 4 Cortex A-75 and 4 Cortex A-55. Task placements and subsequently cpu load will be different for both the platforms. With the same workload, I will expect Rockchip to system to be more loaded than sdm845. Having said that, what cpu-freq governor are you using on both the systems. > > pinebook-pro:~$ cat > /sys/devices/system/cpu/cpufreq/policy4/stats/time_in_state > 408000 16084466 > 600000 27212 > 816000 32487 > 1008000 11331 > 1200000 13268 > 1416000 75078 > 1608000 18392 > 1800000 207266 > 2016000 648612 > > With the Pinebook Pro, which doesn't even come close to getting to 95C, > we spend a lot more time in 2GHz. > > -- steev >
Hi Thara, > Hi Steev, > > IIUC, PineBook Pro has Rockchip RK3399 which has 2 Cortex A-72 and 4 > Cortex A-52 where as C630 has Qualcomm sdm845 which has 4 Cortex A-75 > and 4 Cortex A-55. Task placements and subsequently cpu load will be > different for both the platforms. With the same workload, I will > expect Rockchip to system to be more loaded than sdm845. Having said > that, what cpu-freq governor are you using on both the systems. > I'm using sched-util on both of the systems. I've tried a number of different ways of forcing builds only on the A-75 cores, and I simply cannot get the load to be "enough" to kick in the boost frequency. An example being git clone https://github.com/zellij-org/zellij.git cd zellij taskset --cpu-list 4-7 cargo build --release git clean -fdx taskset --cpu-list 6-7 cargo build --release On my C630, it never goes higher than 85C with the 4 cores being used, and with 2, it never goes about 65C and I do not get any 2.96GHz. It's currently sitting at "6" in the time_in_state for 2965800. --steev
Hi Steev, That's interesting what you've done with Rockchip RK3399. I would like to reproduce your experiment on my RockPI 4B v1.3. Could you tell me how you to add this boost frequency that you have mentioned in some previous emails? I want to have similar setup to yours and I'll check all the subsystems involved in the decision making process for triggering this boost freq. On 11/8/21 11:21 PM, Steev Klimaszewski wrote: > Hi Thara, >> Hi Steev, >> >> IIUC, PineBook Pro has Rockchip RK3399 which has 2 Cortex A-72 and 4 >> Cortex A-52 where as C630 has Qualcomm sdm845 which has 4 Cortex A-75 >> and 4 Cortex A-55. Task placements and subsequently cpu load will be >> different for both the platforms. With the same workload, I will >> expect Rockchip to system to be more loaded than sdm845. Having said >> that, what cpu-freq governor are you using on both the systems. >> > I'm using sched-util on both of the systems. > > I've tried a number of different ways of forcing builds only on the A-75 > cores, and I simply cannot get the load to be "enough" to kick in the > boost frequency. > > An example being > > git clone https://github.com/zellij-org/zellij.git > > cd zellij > > taskset --cpu-list 4-7 cargo build --release > > git clean -fdx > > taskset --cpu-list 6-7 cargo build --release Thanks for the pointers, I'll give it a try when I sort out this Rockchip boost setup. > > > On my C630, it never goes higher than 85C with the 4 cores being used, > and with 2, it never goes about 65C and I do not get any 2.96GHz. It's > currently sitting at "6" in the time_in_state for 2965800. > > > --steev > Thank you for your support. Regards, Lukasz
On 11/9/21 2:29 AM, Lukasz Luba wrote: > Hi Steev, > > That's interesting what you've done with Rockchip RK3399. > I would like to reproduce your experiment on my RockPI 4B v1.3. > Could you tell me how you to add this boost frequency that you have > mentioned in some previous emails? > > I want to have similar setup to yours and I'll check all the subsystems > involved in the decision making process for triggering this boost freq. > > Thank you for your support. > > Regards, > Lukasz Hi Lukasz, It was actually something that Armbian had been doing as an overlay for their setup, and I thought, why does it need to be an overlay, when we could simply hide it behind turbo-mode so that if users want to overclock, they simply echo 1 and if it's unstable or cooling/power isn't enough, they can echo 0 or leave it off (boost defaults to off) - so that being said: I apply this patch https://gitlab.com/kalilinux/build-scripts/kali-arm/-/blob/master/patches/pinebook-pro/pbp-5.14/rk3399-opp-overclock-2GHz-turbo-mode.patch which adds the 1.5GHz for little cores and 2GHz for the big to the rk3399 dtsi To enable at boot time, I simply have "echo 1 > /sys/devices/system/cpu/cpufreq/boost" in my /etc/rc.local And to disable, simply echo 0 in there (it defaults to 0 so it's off and most users won't know it exists.) I'm pretty sure this is "abusing" turbo-mode, but it works well enough... Hope that helps, -- steev
On 11/9/21 3:46 PM, Steev Klimaszewski wrote: > > On 11/9/21 2:29 AM, Lukasz Luba wrote: >> Hi Steev, >> >> That's interesting what you've done with Rockchip RK3399. >> I would like to reproduce your experiment on my RockPI 4B v1.3. >> Could you tell me how you to add this boost frequency that you have >> mentioned in some previous emails? >> >> I want to have similar setup to yours and I'll check all the subsystems >> involved in the decision making process for triggering this boost freq. >> >> Thank you for your support. >> >> Regards, >> Lukasz > > > Hi Lukasz, > > It was actually something that Armbian had been doing as an overlay for > their setup, and I thought, why does it need to be an overlay, when we > could simply hide it behind turbo-mode so that if users want to > overclock, they simply echo 1 and if it's unstable or cooling/power > isn't enough, they can echo 0 or leave it off (boost defaults to off) - > so that being said: > > I apply this patch > https://gitlab.com/kalilinux/build-scripts/kali-arm/-/blob/master/patches/pinebook-pro/pbp-5.14/rk3399-opp-overclock-2GHz-turbo-mode.patch > which adds the 1.5GHz for little cores and 2GHz for the big to the > rk3399 dtsi > > To enable at boot time, I simply have "echo 1 > > /sys/devices/system/cpu/cpufreq/boost" in my /etc/rc.local And to > disable, simply echo 0 in there (it defaults to 0 so it's off and most > users won't know it exists.) > > I'm pretty sure this is "abusing" turbo-mode, but it works well enough... > > Hope that helps, > Yes, that help. Thank you for the info. I'll play a bit with this boosting and try to figure out the mechanisms. For the $subject patch set, I'm going to send v4, since it's not affecting the boost usage. The newly introduced interface must handle these boost frequency values and not simply ignore them with also printing a warning. They are valid frequencies and we should just put 0 to the thermal pressure in such cases. Regards, Lukasz
On 11/9/21 4:22 PM, Lukasz Luba wrote: > > > On 11/9/21 3:46 PM, Steev Klimaszewski wrote: >> >> On 11/9/21 2:29 AM, Lukasz Luba wrote: >>> Hi Steev, >>> >>> That's interesting what you've done with Rockchip RK3399. >>> I would like to reproduce your experiment on my RockPI 4B v1.3. >>> Could you tell me how you to add this boost frequency that you have >>> mentioned in some previous emails? >>> >>> I want to have similar setup to yours and I'll check all the subsystems >>> involved in the decision making process for triggering this boost freq. >>> >>> Thank you for your support. >>> >>> Regards, >>> Lukasz >> >> >> Hi Lukasz, >> >> It was actually something that Armbian had been doing as an overlay >> for their setup, and I thought, why does it need to be an overlay, >> when we could simply hide it behind turbo-mode so that if users want >> to overclock, they simply echo 1 and if it's unstable or cooling/power >> isn't enough, they can echo 0 or leave it off (boost defaults to off) >> - so that being said: >> >> I apply this patch >> https://gitlab.com/kalilinux/build-scripts/kali-arm/-/blob/master/patches/pinebook-pro/pbp-5.14/rk3399-opp-overclock-2GHz-turbo-mode.patch >> which adds the 1.5GHz for little cores and 2GHz for the big to the >> rk3399 dtsi >> >> To enable at boot time, I simply have "echo 1 > >> /sys/devices/system/cpu/cpufreq/boost" in my /etc/rc.local And to >> disable, simply echo 0 in there (it defaults to 0 so it's off and most >> users won't know it exists.) >> >> I'm pretty sure this is "abusing" turbo-mode, but it works well enough... >> >> Hope that helps, >> > > Yes, that help. Thank you for the info. > I'll play a bit with this boosting and try to figure out > the mechanisms. > > For the $subject patch set, I'm going to send v4, since > it's not affecting the boost usage. The newly introduced > interface must handle these boost frequency values and not > simply ignore them with also printing a warning. > They are valid frequencies and we should just put 0 to > the thermal pressure in such cases. > I think I have figure out what is going on with the issue that you've reported. On this rockchip platform you are probably using step-wise thermal governor, which tries to decrease/increase max allowed frequency step-by-step walking through the sorted frequencies. So it would always set the thermal pressure to 0 when the thermal throttling is gone. On the Qcom platform there is a different policy in HW/FW which controls thermal and it can simple remove clamping 'instantly' and allow all frequencies also the boost one. The highest possible frequency is passed then to the this thermal pressure machinery. So we see the warning that the boost frequency value is trying to be passed to this arch_update_thermal_pressure(), but we ignore such big frequency value and unfortunately do not clean the previously set thermal pressure. Then the scheduler still sees the reduced capacity on that CPU and cannot request higher frequencies. The v4 patch would allow to pass the boost frequencies values, so the issue would be solved. Regards, Lukasz
Hi Lukasz, >> > > I think I have figure out what is going on with the issue that > you've reported. On this rockchip platform you are probably using > step-wise thermal governor, which tries to decrease/increase > max allowed frequency step-by-step walking through the sorted > frequencies. So it would always set the thermal pressure to 0 > when the thermal throttling is gone. > On the Qcom platform there is a different policy in HW/FW which > controls thermal and it can simple remove clamping 'instantly' > and allow all frequencies also the boost one. The highest possible > frequency is passed then to the this thermal pressure machinery. > So we see the warning that the boost frequency value is trying to > be passed to this arch_update_thermal_pressure(), but we ignore > such big frequency value and unfortunately do not clean the previously > set thermal pressure. Then the scheduler still sees the reduced > capacity on that CPU and cannot request higher frequencies. > > The v4 patch would allow to pass the boost frequencies values, so > the issue would be solved. > > Regards, > Lukasz Sounds good, I look forward to testing v4 :) -- steev