| Message ID | 20230202215625.3248306-1-usama.arif@bytedance.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Parallel CPU bringup for x86_64 | expand |
On Thu, Feb 02, 2023 at 09:56:14PM +0000, Usama Arif wrote: > Doing INIT/SIPI/SIPI in parallel brings down the time for smpboot from ~700ms > to 100ms (85% improvement) on a server with 128 CPUs split across 2 NUMA nodes. > The parallel CPU bringup is disabled for all AMD CPUs in this version: > (see discussions: https://lore.kernel.org/all/bc3f2b1332c4bb77558df8aa36493a55542fe5b9.camel@infradead.org/ and > https://lore.kernel.org/all/3b6ac86fdc800cac5806433daf14a9095be101e9.camel@infradead.org/). > > The last patch reuses the timer calibration from CPU 0 for secondary CPUs > which brings down the time for parallel smpboot from 100ms to 30ms. It is > missing a sign-off from the author, which hopefully Arjan will add. > > Changes across versions: > v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more > v3: Clean up x2apic patch, add MTRR optimisation, lock topology update > in preparation for more parallelisation. > v4: Fixes to the real mode parallelisation patch spotted by SeanC, to > avoid scribbling on initial_gs in common_cpu_up(), and to allow all > 24 bits of the physical X2APIC ID to be used. That patch still needs > a Signed-off-by from its original author, who once claimed not to > remember writing it at all. But now we've fixed it, hopefully he'll > admit it now :) > v5: rebase to v6.1 and remeasure performance, disable parallel bringup > for AMD CPUs. > v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and > reused timer calibration for secondary CPUs. > > Arjan van de Ven (1): > x86/smpboot: reuse timer calibration > > David Woodhouse (9): > x86/apic/x2apic: Fix parallel handling of cluster_mask > cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> > cpu/hotplug: Add dynamic parallel bringup states before > CPUHP_BRINGUP_CPU > x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() > x86/smpboot: Split up native_cpu_up into separate phases and document > them > x86/smpboot: Disable parallel boot for AMD CPUs > x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel > x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup > x86/smpboot: Serialize topology updates for secondary bringup > > Thomas Gleixner (1): > x86/smpboot: Support parallel startup of secondary CPUs > > arch/x86/include/asm/cpufeatures.h | 1 + > arch/x86/include/asm/realmode.h | 3 + > arch/x86/include/asm/smp.h | 13 +- > arch/x86/include/asm/topology.h | 2 - > arch/x86/kernel/acpi/sleep.c | 1 + > arch/x86/kernel/apic/apic.c | 2 +- > arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++---- > arch/x86/kernel/cpu/amd.c | 11 + > arch/x86/kernel/cpu/common.c | 6 +- > arch/x86/kernel/cpu/mtrr/mtrr.c | 9 + > arch/x86/kernel/head_64.S | 75 ++++++ > arch/x86/kernel/smpboot.c | 333 ++++++++++++++++++-------- > arch/x86/kernel/tsc.c | 3 + > arch/x86/realmode/init.c | 3 + > arch/x86/realmode/rm/trampoline_64.S | 14 ++ > arch/x86/xen/smp_pv.c | 4 +- > include/linux/cpuhotplug.h | 2 + > include/linux/smpboot.h | 7 + > kernel/cpu.c | 27 ++- > kernel/smpboot.c | 2 +- > kernel/smpboot.h | 2 - > 21 files changed, 469 insertions(+), 159 deletions(-) > > -- > 2.25.1 Gave the v6 patchset a try on a system with 1920 logocal cpus (sixteen 60 core Sapphire Rapids sockets with Hyperthreadding enabled). Without the patchset it took 71 seconds to start all the cpus. With the v6 patchset it took 14 seconds to start all the cpus, a reduction of 57 seconds. That is impressive. Full boot, to root login prompt, without patches takes 223 seconds. This patchset reduces the full boot time by 57 seconds, a 25% reduction. Well done. Output with the patchset: -------------------------------------------------------------------------- 2023-02-05 18:12:19Z [ 1.172564][ T1] smp: Bringing up secondary CPUs ... 2023-02-05 18:12:19Z [ 1.179316][ T1] x86: Booting SMP configuration: 2023-02-05 18:12:19Z [ 1.179634][ T1] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 2023-02-05 18:12:20Z [ 1.267650][ T1] .... node #1, CPUs: #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 #96 #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 2023-02-05 18:12:20Z [ 1.315633][ T1] .... node #2, CPUs: #120 #121 #122 #123 #124 #125 #126 #127 #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141 #142 #143 #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156 #157 #158 #159 #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171 #172 #173 #174 #175 #176 #177 #178 #179 2023-02-05 18:12:20Z [ 1.363634][ T1] .... node #3, CPUs: #180 #181 #182 #183 #184 #185 #186 #187 #188 #189 #190 #191 #192 #193 #194 #195 #196 #197 #198 #199 #200 #201 #202 #203 #204 #205 #206 #207 #208 #209 #210 #211 #212 #213 #214 #215 #216 #217 #218 #219 #220 #221 #222 #223 #224 #225 #226 #227 #228 #229 #230 #231 #232 #233 #234 #235 #236 #237 #238 #239 2023-02-05 18:12:20Z [ 1.411634][ T1] .... node #4, CPUs: #240 #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251 #252 #253 #254 #255 #256 #257 #258 #259 #260 #261 #262 #263 #264 #265 #266 #267 #268 #269 #270 #271 #272 #273 #274 #275 #276 #277 #278 #279 #280 #281 #282 #283 #284 #285 #286 #287 #288 #289 #290 #291 #292 #293 #294 #295 #296 #297 #298 #299 2023-02-05 18:12:20Z [ 1.459634][ T1] .... node #5, CPUs: #300 #301 #302 #303 #304 #305 #306 #307 #308 #309 #310 #311 #312 #313 #314 #315 #316 #317 #318 #319 #320 #321 #322 #323 #324 #325 #326 #327 #328 #329 #330 #331 #332 #333 #334 #335 #336 #337 #338 #339 #340 #341 #342 #343 #344 #345 #346 #347 #348 #349 #350 #351 #352 #353 #354 #355 #356 #357 #358 #359 2023-02-05 18:12:20Z [ 1.507634][ T1] .... node #6, CPUs: #360 #361 #362 #363 #364 #365 #366 #367 #368 #369 #370 #371 #372 #373 #374 #375 #376 #377 #378 #379 #380 #381 #382 #383 #384 #385 #386 #387 #388 #389 #390 #391 #392 #393 #394 #395 #396 #397 #398 #399 #400 #401 #402 #403 #404 #405 #406 #407 #408 #409 #410 #411 #412 #413 #414 #415 #416 #417 #418 #419 2023-02-05 18:12:20Z [ 1.555634][ T1] .... node #7, CPUs: #420 #421 #422 #423 #424 #425 #426 #427 #428 #429 #430 #431 #432 #433 #434 #435 #436 #437 #438 #439 #440 #441 #442 #443 #444 #445 #446 #447 #448 #449 #450 #451 #452 #453 #454 #455 #456 #457 #458 #459 #460 #461 #462 #463 #464 #465 #466 #467 #468 #469 #470 #471 #472 #473 #474 #475 #476 #477 #478 #479 2023-02-05 18:12:20Z [ 1.603634][ T1] .... node #8, CPUs: #480 #481 #482 #483 #484 #485 #486 #487 #488 #489 #490 #491 #492 #493 #494 #495 #496 #497 #498 #499 #500 #501 #502 #503 #504 #505 #506 #507 #508 #509 #510 #511 #512 #513 #514 #515 #516 #517 #518 #519 #520 #521 #522 #523 #524 #525 #526 #527 #528 #529 #530 #531 #532 #533 #534 #535 #536 #537 #538 #539 2023-02-05 18:12:20Z [ 1.655634][ T1] .... node #9, CPUs: #540 #541 #542 #543 #544 #545 #546 #547 #548 #549 #550 #551 #552 #553 #554 #555 #556 #557 #558 #559 #560 #561 #562 #563 #564 #565 #566 #567 #568 #569 #570 #571 #572 #573 #574 #575 #576 #577 #578 #579 #580 #581 #582 #583 #584 #585 #586 #587 #588 #589 #590 #591 #592 #593 #594 #595 #596 #597 #598 #599 2023-02-05 18:12:20Z [ 1.707633][ T1] .... node #10, CPUs: #600 #601 #602 #603 #604 #605 #606 #607 #608 #609 #610 #611 #612 #613 #614 #615 #616 #617 #618 #619 #620 #621 #622 #623 #624 #625 #626 #627 #628 #629 #630 #631 #632 #633 #634 #635 #636 #637 #638 #639 #640 #641 #642 #643 #644 #645 #646 #647 #648 #649 #650 #651 #652 #653 #654 #655 #656 #657 #658 #659 2023-02-05 18:12:20Z [ 1.755634][ T1] .... node #11, CPUs: #660 #661 #662 #663 #664 #665 #666 #667 #668 #669 #670 #671 #672 #673 #674 #675 #676 #677 #678 #679 #680 #681 #682 #683 #684 #685 #686 #687 #688 #689 #690 #691 #692 #693 #694 #695 #696 #697 #698 #699 #700 #701 #702 #703 #704 #705 #706 #707 #708 #709 #710 #711 #712 #713 #714 #715 #716 #717 #718 #719 2023-02-05 18:12:20Z [ 1.803634][ T1] .... node #12, CPUs: #720 #721 #722 #723 #724 #725 #726 #727 #728 #729 #730 #731 #732 #733 #734 #735 #736 #737 #738 #739 #740 #741 #742 #743 #744 #745 #746 #747 #748 #749 #750 #751 #752 #753 #754 #755 #756 #757 #758 #759 #760 #761 #762 #763 #764 #765 #766 #767 #768 #769 #770 #771 #772 #773 #774 #775 #776 #777 #778 #779 2023-02-05 18:12:21Z [ 1.855634][ T1] .... node #13, CPUs: #780 #781 #782 #783 #784 #785 #786 #787 #788 #789 #790 #791 #792 #793 #794 #795 #796 #797 #798 #799 #800 #801 #802 #803 #804 #805 #806 #807 #808 #809 #810 #811 #812 #813 #814 #815 #816 #817 #818 #819 #820 #821 #822 #823 #824 #825 #826 #827 #828 #829 #830 #831 #832 #833 #834 #835 #836 #837 #838 #839 2023-02-05 18:12:21Z [ 1.903641][ T1] .... node #14, CPUs: #840 #841 #842 #843 #844 #845 #846 #847 #848 #849 #850 #851 #852 #853 #854 #855 #856 #857 #858 #859 #860 #861 #862 #863 #864 #865 #866 #867 #868 #869 #870 #871 #872 #873 #874 #875 #876 #877 #878 #879 #880 #881 #882 #883 #884 #885 #886 #887 #888 #889 #890 #891 #892 #893 #894 #895 #896 #897 #898 #899 2023-02-05 18:12:21Z [ 1.951634][ T1] .... node #15, CPUs: #900 #901 #902 #903 #904 #905 #906 #907 #908 #909 #910 #911 #912 #913 #914 #915 #916 #917 #918 #919 #920 #921 #922 #923 #924 #925 #926 #927 #928 #929 #930 #931 #932 #933 #934 #935 #936 #937 #938 #939 #940 #941 #942 #943 #944 #945 #946 #947 #948 #949 #950 #951 #952 #953 #954 #955 #956 #957 #958 #959 2023-02-05 18:12:21Z [ 1.999634][ T1] .... node #0, CPUs: #960 #961 #962 #963 #964 #965 #966 #967 #968 #969 #970 #971 #972 #973 #974 #975 #976 #977 #978 #979 #980 #981 #982 #983 #984 #985 #986 #987 #988 #989 #990 #991 #992 #993 #994 #995 #996 #997 #998 #999 #1000 #1001 #1002 #1003 #1004 #1005 #1006 #1007 #1008 #1009 #1010 #1011 #1012 #1013 #1014 #1015 #1016 #1017 #1018 #1019 2023-02-05 18:12:21Z [ 2.079642][ T1] .... node #1, CPUs: #1020 #1021 #1022 #1023 #1024 #1025 #1026 #1027 #1028 #1029 #1030 #1031 #1032 #1033 #1034 #1035 #1036 #1037 #1038 #1039 #1040 #1041 #1042 #1043 #1044 #1045 #1046 #1047 #1048 #1049 #1050 #1051 #1052 #1053 #1054 #1055 #1056 #1057 #1058 #1059 #1060 #1061 #1062 #1063 #1064 #1065 #1066 #1067 #1068 #1069 #1070 #1071 #1072 #1073 #1074 #1075 #1076 #1077 #1078 #1079 2023-02-05 18:12:21Z [ 2.119634][ T1] .... node #2, CPUs: #1080 #1081 #1082 #1083 #1084 #1085 #1086 #1087 #1088 #1089 #1090 #1091 #1092 #1093 #1094 #1095 #1096 #1097 #1098 #1099 #1100 #1101 #1102 #1103 #1104 #1105 #1106 #1107 #1108 #1109 #1110 #1111 #1112 #1113 #1114 #1115 #1116 #1117 #1118 #1119 #1120 #1121 #1122 #1123 #1124 #1125 #1126 #1127 #1128 #1129 #1130 #1131 #1132 #1133 #1134 #1135 #1136 #1137 #1138 #1139 2023-02-05 18:12:21Z [ 2.155634][ T1] .... node #3, CPUs: #1140 #1141 #1142 #1143 #1144 #1145 #1146 #1147 #1148 #1149 #1150 #1151 #1152 #1153 #1154 #1155 #1156 #1157 #1158 #1159 #1160 #1161 #1162 #1163 #1164 #1165 #1166 #1167 #1168 #1169 #1170 #1171 #1172 #1173 #1174 #1175 #1176 #1177 #1178 #1179 #1180 #1181 #1182 #1183 #1184 #1185 #1186 #1187 #1188 #1189 #1190 #1191 #1192 #1193 #1194 #1195 #1196 #1197 #1198 #1199 2023-02-05 18:12:21Z [ 2.195634][ T1] .... node #4, CPUs: #1200 #1201 #1202 #1203 #1204 #1205 #1206 #1207 #1208 #1209 #1210 #1211 #1212 #1213 #1214 #1215 #1216 #1217 #1218 #1219 #1220 #1221 #1222 #1223 #1224 #1225 #1226 #1227 #1228 #1229 #1230 #1231 #1232 #1233 #1234 #1235 #1236 #1237 #1238 #1239 #1240 #1241 #1242 #1243 #1244 #1245 #1246 #1247 #1248 #1249 #1250 #1251 #1252 #1253 #1254 #1255 #1256 #1257 #1258 #1259 2023-02-05 18:12:21Z [ 2.235635][ T1] .... node #5, CPUs: #1260 #1261 #1262 #1263 #1264 #1265 #1266 #1267 #1268 #1269 #1270 #1271 #1272 #1273 #1274 #1275 #1276 #1277 #1278 #1279 #1280 #1281 #1282 #1283 #1284 #1285 #1286 #1287 #1288 #1289 #1290 #1291 #1292 #1293 #1294 #1295 #1296 #1297 #1298 #1299 #1300 #1301 #1302 #1303 #1304 #1305 #1306 #1307 #1308 #1309 #1310 #1311 #1312 #1313 #1314 #1315 #1316 #1317 #1318 #1319 2023-02-05 18:12:21Z [ 2.275634][ T1] .... node #6, CPUs: #1320 #1321 #1322 #1323 #1324 #1325 #1326 #1327 #1328 #1329 #1330 #1331 #1332 #1333 #1334 #1335 #1336 #1337 #1338 #1339 #1340 #1341 #1342 #1343 #1344 #1345 #1346 #1347 #1348 #1349 #1350 #1351 #1352 #1353 #1354 #1355 #1356 #1357 #1358 #1359 #1360 #1361 #1362 #1363 #1364 #1365 #1366 #1367 #1368 #1369 #1370 #1371 #1372 #1373 #1374 #1375 #1376 #1377 #1378 #1379 2023-02-05 18:12:21Z [ 2.320561][ T1] .... node #7, CPUs: #1380 #1381 #1382 #1383 #1384 #1385 #1386 #1387 #1388 #1389 #1390 #1391 #1392 #1393 #1394 #1395 #1396 #1397 #1398 #1399 #1400 #1401 #1402 #1403 #1404 #1405 #1406 #1407 #1408 #1409 #1410 #1411 #1412 #1413 #1414 #1415 #1416 #1417 #1418 #1419 #1420 #1421 #1422 #1423 #1424 #1425 #1426 #1427 #1428 #1429 #1430 #1431 #1432 #1433 #1434 #1435 #1436 #1437 #1438 #1439 2023-02-05 18:12:21Z [ 2.363647][ T1] .... node #8, CPUs: #1440 #1441 #1442 #1443 #1444 #1445 #1446 #1447 #1448 #1449 #1450 #1451 #1452 #1453 #1454 #1455 #1456 #1457 #1458 #1459 #1460 #1461 #1462 #1463 #1464 #1465 #1466 #1467 #1468 #1469 #1470 #1471 #1472 #1473 #1474 #1475 #1476 #1477 #1478 #1479 #1480 #1481 #1482 #1483 #1484 #1485 #1486 #1487 #1488 #1489 #1490 #1491 #1492 #1493 #1494 #1495 #1496 #1497 #1498 #1499 2023-02-05 18:12:22Z [ 2.407636][ T1] .... node #9, CPUs: #1500 #1501 #1502 #1503 #1504 #1505 #1506 #1507 #1508 #1509 #1510 #1511 #1512 #1513 #1514 #1515 #1516 #1517 #1518 #1519 #1520 #1521 #1522 #1523 #1524 #1525 #1526 #1527 #1528 #1529 #1530 #1531 #1532 #1533 #1534 #1535 #1536 #1537 #1538 #1539 #1540 #1541 #1542 #1543 #1544 #1545 #1546 #1547 #1548 #1549 #1550 #1551 #1552 #1553 #1554 #1555 #1556 #1557 #1558 #1559 2023-02-05 18:12:22Z [ 2.451644][ T1] .... node #10, CPUs: #1560 #1561 #1562 #1563 #1564 #1565 #1566 #1567 #1568 #1569 #1570 #1571 #1572 #1573 #1574 #1575 #1576 #1577 #1578 #1579 #1580 #1581 #1582 #1583 #1584 #1585 #1586 #1587 #1588 #1589 #1590 #1591 #1592 #1593 #1594 #1595 #1596 #1597 #1598 #1599 #1600 #1601 #1602 #1603 #1604 #1605 #1606 #1607 #1608 #1609 #1610 #1611 #1612 #1613 #1614 #1615 #1616 #1617 #1618 #1619 2023-02-05 18:12:22Z [ 2.495648][ T1] .... node #11, CPUs: #1620 #1621 #1622 #1623 #1624 #1625 #1626 #1627 #1628 #1629 #1630 #1631 #1632 #1633 #1634 #1635 #1636 #1637 #1638 #1639 #1640 #1641 #1642 #1643 #1644 #1645 #1646 #1647 #1648 #1649 #1650 #1651 #1652 #1653 #1654 #1655 #1656 #1657 #1658 #1659 #1660 #1661 #1662 #1663 #1664 #1665 #1666 #1667 #1668 #1669 #1670 #1671 #1672 #1673 #1674 #1675 #1676 #1677 #1678 #1679 2023-02-05 18:12:22Z [ 2.539634][ T1] .... node #12, CPUs: #1680 #1681 #1682 #1683 #1684 #1685 #1686 #1687 #1688 #1689 #1690 #1691 #1692 #1693 #1694 #1695 #1696 #1697 #1698 #1699 #1700 #1701 #1702 #1703 #1704 #1705 #1706 #1707 #1708 #1709 #1710 #1711 #1712 #1713 #1714 #1715 #1716 #1717 #1718 #1719 #1720 #1721 #1722 #1723 #1724 #1725 #1726 #1727 #1728 #1729 #1730 #1731 #1732 #1733 #1734 #1735 #1736 #1737 #1738 #1739 2023-02-05 18:12:22Z [ 2.583644][ T1] .... node #13, CPUs: #1740 #1741 #1742 #1743 #1744 #1745 #1746 #1747 #1748 #1749 #1750 #1751 #1752 #1753 #1754 #1755 #1756 #1757 #1758 #1759 #1760 #1761 #1762 #1763 #1764 #1765 #1766 #1767 #1768 #1769 #1770 #1771 #1772 #1773 #1774 #1775 #1776 #1777 #1778 #1779 #1780 #1781 #1782 #1783 #1784 #1785 #1786 #1787 #1788 #1789 #1790 #1791 #1792 #1793 #1794 #1795 #1796 #1797 #1798 #1799 2023-02-05 18:12:22Z [ 2.627635][ T1] .... node #14, CPUs: #1800 #1801 #1802 #1803 #1804 #1805 #1806 #1807 #1808 #1809 #1810 #1811 #1812 #1813 #1814 #1815 #1816 #1817 #1818 #1819 #1820 #1821 #1822 #1823 #1824 #1825 #1826 #1827 #1828 #1829 #1830 #1831 #1832 #1833 #1834 #1835 #1836 #1837 #1838 #1839 #1840 #1841 #1842 #1843 #1844 #1845 #1846 #1847 #1848 #1849 #1850 #1851 #1852 #1853 #1854 #1855 #1856 #1857 #1858 #1859 2023-02-05 18:12:22Z [ 2.675650][ T1] .... node #15, CPUs: #1860 #1861 #1862 #1863 #1864 #1865 #1866 #1867 #1868 #1869 #1870 #1871 #1872 #1873 #1874 #1875 #1876 #1877 #1878 #1879 #1880 #1881 #1882 #1883 #1884 #1885 #1886 #1887 #1888 #1889 #1890 #1891 #1892 #1893 #1894 #1895 #1896 #1897 #1898 #1899 #1900 #1901 #1902 #1903 #1904 #1905 #1906 #1907 #1908 #1909 #1910 #1911 #1912 #1913 #1914 #1915 #1916 #1917 #1918 #1919 2023-02-05 18:12:22Z [ 0.000000][ T0] smpboot: CPU 60 Converting physical 0 to logical die 1 2023-02-05 18:12:22Z [ 0.000000][ T0] smpboot: CPU 120 Converting physical 0 to logical die 2 2023-02-05 18:12:22Z [ 0.000000][ T0] smpboot: CPU 180 Converting physical 0 to logical die 3 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 240 Converting physical 0 to logical die 4 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 300 Converting physical 0 to logical die 5 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 360 Converting physical 0 to logical die 6 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 420 Converting physical 0 to logical die 7 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 480 Converting physical 0 to logical die 8 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 540 Converting physical 0 to logical die 9 2023-02-05 18:12:24Z [ 0.000000][ T0] smpboot: CPU 600 Converting physical 0 to logical die 10 2023-02-05 18:12:24Z [ 0.000000][ T0] smpboot: CPU 660 Converting physical 0 to logical die 11 2023-02-05 18:12:24Z [ 0.000000][ T0] smpboot: CPU 720 Converting physical 0 to logical die 12 2023-02-05 18:12:25Z [ 0.000000][ T0] smpboot: CPU 780 Converting physical 0 to logical die 13 2023-02-05 18:12:25Z [ 0.000000][ T0] smpboot: CPU 840 Converting physical 0 to logical die 14 2023-02-05 18:12:25Z [ 0.000000][ T0] smpboot: CPU 900 Converting physical 0 to logical die 15 2023-02-05 18:12:35Z [ 15.213189][ T1] smp: Brought up 16 nodes, 1920 CPUs 2023-02-05 18:12:35Z [ 15.215637][ T1] smpboot: Max logical packages: 16 2023-02-05 18:12:35Z [ 15.219750][ T1] smpboot: Total of 1920 processors activated (7296000.00 BogoMIPS) -------------------------------------------------------------------------- Output without the patchset: -------------------------------------------------------------------------- 2023-02-03 11:06:49Z [ 1.157637][ T1] smp: Bringing up secondary CPUs ... 2023-02-03 11:06:49Z [ 1.162807][ T1] x86: Booting SMP configuration: 2023-02-03 11:06:51Z [ 1.164299][ T1] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 2023-02-03 11:06:51Z [ 3.052299][ T1] .... node #1, CPUs: #60 2023-02-03 11:06:51Z [ 0.000000][ T0] smpboot: CPU 60 Converting physical 0 to logical die 1 2023-02-03 11:06:53Z [ 3.133499][ T1] #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 #96 #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 2023-02-03 11:06:53Z [ 4.984298][ T1] .... node #2, CPUs: #120 2023-02-03 11:06:53Z [ 0.000000][ T0] smpboot: CPU 120 Converting physical 0 to logical die 2 2023-02-03 11:06:55Z [ 5.065813][ T1] #121 #122 #123 #124 #125 #126 #127 #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141 #142 #143 #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156 #157 #158 #159 #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171 #172 #173 #174 #175 #176 #177 #178 #179 2023-02-03 11:06:55Z [ 6.920298][ T1] .... node #3, CPUs: #180 2023-02-03 11:06:55Z [ 0.000000][ T0] smpboot: CPU 180 Converting physical 0 to logical die 3 2023-02-03 11:06:57Z [ 7.002019][ T1] #181 #182 #183 #184 #185 #186 #187 #188 #189 #190 #191 #192 #193 #194 #195 #196 #197 #198 #199 #200 #201 #202 #203 #204 #205 #206 #207 #208 #209 #210 #211 #212 #213 #214 #215 #216 #217 #218 #219 #220 #221 #222 #223 #224 #225 #226 #227 #228 #229 #230 #231 #232 #233 #234 #235 #236 #237 #238 #239 2023-02-03 11:06:57Z [ 8.880299][ T1] .... node #4, CPUs: #240 2023-02-03 11:06:57Z [ 0.000000][ T0] smpboot: CPU 240 Converting physical 0 to logical die 4 2023-02-03 11:06:59Z [ 8.991204][ T1] #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251 #252 #253 #254 #255 #256 #257 #258 #259 #260 #261 #262 #263 #264 #265 #266 #267 #268 #269 #270 #271 #272 #273 #274 #275 #276 #277 #278 #279 #280 #281 #282 #283 #284 #285 #286 #287 #288 #289 #290 #291 #292 #293 #294 #295 #296 #297 #298 #299 2023-02-03 11:06:59Z [ 10.944299][ T1] .... node #5, CPUs: #300 2023-02-03 11:06:59Z [ 0.000000][ T0] smpboot: CPU 300 Converting physical 0 to logical die 5 2023-02-03 11:07:01Z [ 11.027023][ T1] #301 #302 #303 #304 #305 #306 #307 #308 #309 #310 #311 #312 #313 #314 #315 #316 #317 #318 #319 #320 #321 #322 #323 #324 #325 #326 #327 #328 #329 #330 #331 #332 #333 #334 #335 #336 #337 #338 #339 #340 #341 #342 #343 #344 #345 #346 #347 #348 #349 #350 #351 #352 #353 #354 #355 #356 #357 #358 #359 2023-02-03 11:07:01Z [ 12.964299][ T1] .... node #6, CPUs: #360 2023-02-03 11:07:01Z [ 0.000000][ T0] smpboot: CPU 360 Converting physical 0 to logical die 6 2023-02-03 11:07:03Z [ 13.047169][ T1] #361 #362 #363 #364 #365 #366 #367 #368 #369 #370 #371 #372 #373 #374 #375 #376 #377 #378 #379 #380 #381 #382 #383 #384 #385 #386 #387 #388 #389 #390 #391 #392 #393 #394 #395 #396 #397 #398 #399 #400 #401 #402 #403 #404 #405 #406 #407 #408 #409 #410 #411 #412 #413 #414 #415 #416 #417 #418 #419 2023-02-03 11:07:03Z [ 14.996299][ T1] .... node #7, CPUs: #420 2023-02-03 11:07:03Z [ 0.000000][ T0] smpboot: CPU 420 Converting physical 0 to logical die 7 2023-02-03 11:07:05Z [ 15.079323][ T1] #421 #422 #423 #424 #425 #426 #427 #428 #429 #430 #431 #432 #433 #434 #435 #436 #437 #438 #439 #440 #441 #442 #443 #444 #445 #446 #447 #448 #449 #450 #451 #452 #453 #454 #455 #456 #457 #458 #459 #460 #461 #462 #463 #464 #465 #466 #467 #468 #469 #470 #471 #472 #473 #474 #475 #476 #477 #478 #479 2023-02-03 11:07:06Z [ 17.048299][ T1] .... node #8, CPUs: #480 2023-02-03 11:07:06Z [ 0.000000][ T0] smpboot: CPU 480 Converting physical 0 to logical die 8 2023-02-03 11:07:08Z [ 17.161077][ T1] #481 #482 #483 #484 #485 #486 #487 #488 #489 #490 #491 #492 #493 #494 #495 #496 #497 #498 #499 #500 #501 #502 #503 #504 #505 #506 #507 #508 #509 #510 #511 #512 #513 #514 #515 #516 #517 #518 #519 #520 #521 #522 #523 #524 #525 #526 #527 #528 #529 #530 #531 #532 #533 #534 #535 #536 #537 #538 #539 2023-02-03 11:07:08Z [ 19.196298][ T1] .... node #9, CPUs: #540 2023-02-03 11:07:08Z [ 0.000000][ T0] smpboot: CPU 540 Converting physical 0 to logical die 9 2023-02-03 11:07:10Z [ 19.281098][ T1] #541 #542 #543 #544 #545 #546 #547 #548 #549 #550 #551 #552 #553 #554 #555 #556 #557 #558 #559 #560 #561 #562 #563 #564 #565 #566 #567 #568 #569 #570 #571 #572 #573 #574 #575 #576 #577 #578 #579 #580 #581 #582 #583 #584 #585 #586 #587 #588 #589 #590 #591 #592 #593 #594 #595 #596 #597 #598 #599 2023-02-03 11:07:10Z [ 21.324298][ T1] .... node #10, CPUs: #600 2023-02-03 11:07:10Z [ 0.000000][ T0] smpboot: CPU 600 Converting physical 0 to logical die 10 2023-02-03 11:07:12Z [ 21.409074][ T1] #601 #602 #603 #604 #605 #606 #607 #608 #609 #610 #611 #612 #613 #614 #615 #616 #617 #618 #619 #620 #621 #622 #623 #624 #625 #626 #627 #628 #629 #630 #631 #632 #633 #634 #635 #636 #637 #638 #639 #640 #641 #642 #643 #644 #645 #646 #647 #648 #649 #650 #651 #652 #653 #654 #655 #656 #657 #658 #659 2023-02-03 11:07:12Z [ 23.468298][ T1] .... node #11, CPUs: #660 2023-02-03 11:07:12Z [ 0.000000][ T0] smpboot: CPU 660 Converting physical 0 to logical die 11 2023-02-03 11:07:14Z [ 23.553115][ T1] #661 #662 #663 #664 #665 #666 #667 #668 #669 #670 #671 #672 #673 #674 #675 #676 #677 #678 #679 #680 #681 #682 #683 #684 #685 #686 #687 #688 #689 #690 #691 #692 #693 #694 #695 #696 #697 #698 #699 #700 #701 #702 #703 #704 #705 #706 #707 #708 #709 #710 #711 #712 #713 #714 #715 #716 #717 #718 #719 2023-02-03 11:07:14Z [ 25.624298][ T1] .... node #12, CPUs: #720 2023-02-03 11:07:14Z [ 0.000000][ T0] smpboot: CPU 720 Converting physical 0 to logical die 12 2023-02-03 11:07:17Z [ 25.741550][ T1] #721 #722 #723 #724 #725 #726 #727 #728 #729 #730 #731 #732 #733 #734 #735 #736 #737 #738 #739 #740 #741 #742 #743 #744 #745 #746 #747 #748 #749 #750 #751 #752 #753 #754 #755 #756 #757 #758 #759 #760 #761 #762 #763 #764 #765 #766 #767 #768 #769 #770 #771 #772 #773 #774 #775 #776 #777 #778 #779 2023-02-03 11:07:17Z [ 27.876299][ T1] .... node #13, CPUs: #780 2023-02-03 11:07:17Z [ 0.000000][ T0] smpboot: CPU 780 Converting physical 0 to logical die 13 2023-02-03 11:07:19Z [ 27.962255][ T1] #781 #782 #783 #784 #785 #786 #787 #788 #789 #790 #791 #792 #793 #794 #795 #796 #797 #798 #799 #800 #801 #802 #803 #804 #805 #806 #807 #808 #809 #810 #811 #812 #813 #814 #815 #816 #817 #818 #819 #820 #821 #822 #823 #824 #825 #826 #827 #828 #829 #830 #831 #832 #833 #834 #835 #836 #837 #838 #839 2023-02-03 11:07:19Z [ 30.096299][ T1] .... node #14, CPUs: #840 2023-02-03 11:07:19Z [ 0.000000][ T0] smpboot: CPU 840 Converting physical 0 to logical die 14 2023-02-03 11:07:21Z [ 30.182443][ T1] #841 #842 #843 #844 #845 #846 #847 #848 #849 #850 #851 #852 #853 #854 #855 #856 #857 #858 #859 #860 #861 #862 #863 #864 #865 #866 #867 #868 #869 #870 #871 #872 #873 #874 #875 #876 #877 #878 #879 #880 #881 #882 #883 #884 #885 #886 #887 #888 #889 #890 #891 #892 #893 #894 #895 #896 #897 #898 #899 2023-02-03 11:07:21Z [ 32.332299][ T1] .... node #15, CPUs: #900 2023-02-03 11:07:21Z [ 0.000000][ T0] smpboot: CPU 900 Converting physical 0 to logical die 15 2023-02-03 11:07:23Z [ 32.420700][ T1] #901 #902 #903 #904 #905 #906 #907 #908 #909 #910 #911 #912 #913 #914 #915 #916 #917 #918 #919 #920 #921 #922 #923 #924 #925 #926 #927 #928 #929 #930 #931 #932 #933 #934 #935 #936 #937 #938 #939 #940 #941 #942 #943 #944 #945 #946 #947 #948 #949 #950 #951 #952 #953 #954 #955 #956 #957 #958 #959 2023-02-03 11:07:26Z [ 34.584298][ T1] .... node #0, CPUs: #960 #961 #962 #963 #964 #965 #966 #967 #968 #969 #970 #971 #972 #973 #974 #975 #976 #977 #978 #979 #980 #981 #982 #983 #984 #985 #986 #987 #988 #989 #990 #991 #992 #993 #994 #995 #996 #997 #998 #999 #1000 #1001 #1002 #1003 #1004 #1005 #1006 #1007 #1008 #1009 #1010 #1011 #1012 #1013 #1014 #1015 #1016 #1017 #1018 #1019 2023-02-03 11:07:28Z [ 36.792301][ T1] .... node #1, CPUs: #1020 #1021 #1022 #1023 #1024 #1025 #1026 #1027 #1028 #1029 #1030 #1031 #1032 #1033 #1034 #1035 #1036 #1037 #1038 #1039 #1040 #1041 #1042 #1043 #1044 #1045 #1046 #1047 #1048 #1049 #1050 #1051 #1052 #1053 #1054 #1055 #1056 #1057 #1058 #1059 #1060 #1061 #1062 #1063 #1064 #1065 #1066 #1067 #1068 #1069 #1070 #1071 #1072 #1073 #1074 #1075 #1076 #1077 #1078 #1079 2023-02-03 11:07:30Z [ 38.980298][ T1] .... node #2, CPUs: #1080 #1081 #1082 #1083 #1084 #1085 #1086 #1087 #1088 #1089 #1090 #1091 #1092 #1093 #1094 #1095 #1096 #1097 #1098 #1099 #1100 #1101 #1102 #1103 #1104 #1105 #1106 #1107 #1108 #1109 #1110 #1111 #1112 #1113 #1114 #1115 #1116 #1117 #1118 #1119 #1120 #1121 #1122 #1123 #1124 #1125 #1126 #1127 #1128 #1129 #1130 #1131 #1132 #1133 #1134 #1135 #1136 #1137 #1138 #1139 2023-02-03 11:07:32Z [ 41.180298][ T1] .... node #3, CPUs: #1140 #1141 #1142 #1143 #1144 #1145 #1146 #1147 #1148 #1149 #1150 #1151 #1152 #1153 #1154 #1155 #1156 #1157 #1158 #1159 #1160 #1161 #1162 #1163 #1164 #1165 #1166 #1167 #1168 #1169 #1170 #1171 #1172 #1173 #1174 #1175 #1176 #1177 #1178 #1179 #1180 #1181 #1182 #1183 #1184 #1185 #1186 #1187 #1188 #1189 #1190 #1191 #1192 #1193 #1194 #1195 #1196 #1197 #1198 #1199 2023-02-03 11:07:35Z [ 43.408300][ T1] .... node #4, CPUs: #1200 #1201 #1202 #1203 #1204 #1205 #1206 #1207 #1208 #1209 #1210 #1211 #1212 #1213 #1214 #1215 #1216 #1217 #1218 #1219 #1220 #1221 #1222 #1223 #1224 #1225 #1226 #1227 #1228 #1229 #1230 #1231 #1232 #1233 #1234 #1235 #1236 #1237 #1238 #1239 #1240 #1241 #1242 #1243 #1244 #1245 #1246 #1247 #1248 #1249 #1250 #1251 #1252 #1253 #1254 #1255 #1256 #1257 #1258 #1259 2023-02-03 11:07:37Z [ 45.744298][ T1] .... node #5, CPUs: #1260 #1261 #1262 #1263 #1264 #1265 #1266 #1267 #1268 #1269 #1270 #1271 #1272 #1273 #1274 #1275 #1276 #1277 #1278 #1279 #1280 #1281 #1282 #1283 #1284 #1285 #1286 #1287 #1288 #1289 #1290 #1291 #1292 #1293 #1294 #1295 #1296 #1297 #1298 #1299 #1300 #1301 #1302 #1303 #1304 #1305 #1306 #1307 #1308 #1309 #1310 #1311 #1312 #1313 #1314 #1315 #1316 #1317 #1318 #1319 2023-02-03 11:07:39Z [ 48.056299][ T1] .... node #6, CPUs: #1320 #1321 #1322 #1323 #1324 #1325 #1326 #1327 #1328 #1329 #1330 #1331 #1332 #1333 #1334 #1335 #1336 #1337 #1338 #1339 #1340 #1341 #1342 #1343 #1344 #1345 #1346 #1347 #1348 #1349 #1350 #1351 #1352 #1353 #1354 #1355 #1356 #1357 #1358 #1359 #1360 #1361 #1362 #1363 #1364 #1365 #1366 #1367 #1368 #1369 #1370 #1371 #1372 #1373 #1374 #1375 #1376 #1377 #1378 #1379 2023-02-03 11:07:42Z [ 50.380298][ T1] .... node #7, CPUs: #1380 #1381 #1382 #1383 #1384 #1385 #1386 #1387 #1388 #1389 #1390 #1391 #1392 #1393 #1394 #1395 #1396 #1397 #1398 #1399 #1400 #1401 #1402 #1403 #1404 #1405 #1406 #1407 #1408 #1409 #1410 #1411 #1412 #1413 #1414 #1415 #1416 #1417 #1418 #1419 #1420 #1421 #1422 #1423 #1424 #1425 #1426 #1427 #1428 #1429 #1430 #1431 #1432 #1433 #1434 #1435 #1436 #1437 #1438 #1439 2023-02-03 11:07:44Z [ 52.736298][ T1] .... node #8, CPUs: #1440 #1441 #1442 #1443 #1444 #1445 #1446 #1447 #1448 #1449 #1450 #1451 #1452 #1453 #1454 #1455 #1456 #1457 #1458 #1459 #1460 #1461 #1462 #1463 #1464 #1465 #1466 #1467 #1468 #1469 #1470 #1471 #1472 #1473 #1474 #1475 #1476 #1477 #1478 #1479 #1480 #1481 #1482 #1483 #1484 #1485 #1486 #1487 #1488 #1489 #1490 #1491 #1492 #1493 #1494 #1495 #1496 #1497 #1498 #1499 2023-02-03 11:07:47Z [ 55.164301][ T1] .... node #9, CPUs: #1500 #1501 #1502 #1503 #1504 #1505 #1506 #1507 #1508 #1509 #1510 #1511 #1512 #1513 #1514 #1515 #1516 #1517 #1518 #1519 #1520 #1521 #1522 #1523 #1524 #1525 #1526 #1527 #1528 #1529 #1530 #1531 #1532 #1533 #1534 #1535 #1536 #1537 #1538 #1539 #1540 #1541 #1542 #1543 #1544 #1545 #1546 #1547 #1548 #1549 #1550 #1551 #1552 #1553 #1554 #1555 #1556 #1557 #1558 #1559 2023-02-03 11:07:49Z [ 57.592298][ T1] .... node #10, CPUs: #1560 #1561 #1562 #1563 #1564 #1565 #1566 #1567 #1568 #1569 #1570 #1571 #1572 #1573 #1574 #1575 #1576 #1577 #1578 #1579 #1580 #1581 #1582 #1583 #1584 #1585 #1586 #1587 #1588 #1589 #1590 #1591 #1592 #1593 #1594 #1595 #1596 #1597 #1598 #1599 #1600 #1601 #1602 #1603 #1604 #1605 #1606 #1607 #1608 #1609 #1610 #1611 #1612 #1613 #1614 #1615 #1616 #1617 #1618 #1619 2023-02-03 11:07:52Z [ 60.032299][ T1] .... node #11, CPUs: #1620 #1621 #1622 #1623 #1624 #1625 #1626 #1627 #1628 #1629 #1630 #1631 #1632 #1633 #1634 #1635 #1636 #1637 #1638 #1639 #1640 #1641 #1642 #1643 #1644 #1645 #1646 #1647 #1648 #1649 #1650 #1651 #1652 #1653 #1654 #1655 #1656 #1657 #1658 #1659 #1660 #1661 #1662 #1663 #1664 #1665 #1666 #1667 #1668 #1669 #1670 #1671 #1672 #1673 #1674 #1675 #1676 #1677 #1678 #1679 2023-02-03 11:07:54Z [ 62.484298][ T1] .... node #12, CPUs: #1680 #1681 #1682 #1683 #1684 #1685 #1686 #1687 #1688 #1689 #1690 #1691 #1692 #1693 #1694 #1695 #1696 #1697 #1698 #1699 #1700 #1701 #1702 #1703 #1704 #1705 #1706 #1707 #1708 #1709 #1710 #1711 #1712 #1713 #1714 #1715 #1716 #1717 #1718 #1719 #1720 #1721 #1722 #1723 #1724 #1725 #1726 #1727 #1728 #1729 #1730 #1731 #1732 #1733 #1734 #1735 #1736 #1737 #1738 #1739 2023-02-03 11:07:57Z [ 65.024298][ T1] .... node #13, CPUs: #1740 #1741 #1742 #1743 #1744 #1745 #1746 #1747 #1748 #1749 #1750 #1751 #1752 #1753 #1754 #1755 #1756 #1757 #1758 #1759 #1760 #1761 #1762 #1763 #1764 #1765 #1766 #1767 #1768 #1769 #1770 #1771 #1772 #1773 #1774 #1775 #1776 #1777 #1778 #1779 #1780 #1781 #1782 #1783 #1784 #1785 #1786 #1787 #1788 #1789 #1790 #1791 #1792 #1793 #1794 #1795 #1796 #1797 #1798 #1799 2023-02-03 11:07:59Z [ 67.544299][ T1] .... node #14, CPUs: #1800 #1801 #1802 #1803 #1804 #1805 #1806 #1807 #1808 #1809 #1810 #1811 #1812 #1813 #1814 #1815 #1816 #1817 #1818 #1819 #1820 #1821 #1822 #1823 #1824 #1825 #1826 #1827 #1828 #1829 #1830 #1831 #1832 #1833 #1834 #1835 #1836 #1837 #1838 #1839 #1840 #1841 #1842 #1843 #1844 #1845 #1846 #1847 #1848 #1849 #1850 #1851 #1852 #1853 #1854 #1855 #1856 #1857 #1858 #1859 2023-02-03 11:08:02Z [ 70.084299][ T1] .... node #15, CPUs: #1860 #1861 #1862 #1863 #1864 #1865 #1866 #1867 #1868 #1869 #1870 #1871 #1872 #1873 #1874 #1875 #1876 #1877 #1878 #1879 #1880 #1881 #1882 #1883 #1884 #1885 #1886 #1887 #1888 #1889 #1890 #1891 #1892 #1893 #1894 #1895 #1896 #1897 #1898 #1899 #1900 #1901 #1902 #1903 #1904 #1905 #1906 #1907 #1908 #1909 #1910 #1911 #1912 #1913 #1914 #1915 #1916 #1917 #1918 #1919 2023-02-03 11:08:02Z [ 72.634158][ T1] smp: Brought up 16 nodes, 1920 CPUs 2023-02-03 11:08:02Z [ 72.640299][ T1] smpboot: Max logical packages: 16 2023-02-03 11:08:02Z [ 72.644444][ T1] smpboot: Total of 1920 processors activated (7275848.16 BogoMIPS) -------------------------------------------------------------------------- Thanks.
On Sun, 2023-02-05 at 13:17 -0600, Russ Anderson wrote: > > Gave the v6 patchset a try on a system with 1920 logocal cpus > (sixteen 60 core Sapphire Rapids sockets with Hyperthreadding > enabled). > > Without the patchset it took 71 seconds to start all the cpus. > With the v6 patchset it took 14 seconds to start all the cpus, > a reduction of 57 seconds. That is impressive. > > Full boot, to root login prompt, without patches takes 223 seconds. > This patchset reduces the full boot time by 57 seconds, a 25% > reduction. Nice; thanks for testing. Is that with just the "part1" patch series which has been posted, or also with the 'parallel part 2' still taking shape in the tree at https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/parallel-6.2-rc6 I believe Usama said the second phase of parallelism didn't really help much in terms of overall timing? Confirming that *without* all the debug prints would be interesting. And we can look for what still *could* be made parallel.
On 06/02/2023 08:28, David Woodhouse wrote: > On Sun, 2023-02-05 at 13:17 -0600, Russ Anderson wrote: >> >> Gave the v6 patchset a try on a system with 1920 logocal cpus >> (sixteen 60 core Sapphire Rapids sockets with Hyperthreadding >> enabled). >> >> Without the patchset it took 71 seconds to start all the cpus. >> With the v6 patchset it took 14 seconds to start all the cpus, >> a reduction of 57 seconds. That is impressive. >> >> Full boot, to root login prompt, without patches takes 223 seconds. >> This patchset reduces the full boot time by 57 seconds, a 25% >> reduction. > > Nice; thanks for testing. > > Is that with just the "part1" patch series which has been posted, or > also with the 'parallel part 2' still taking shape in the tree at > https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/parallel-6.2-rc6 > > I believe Usama said the second phase of parallelism didn't really help > much in terms of overall timing? Confirming that *without* all the > debug prints would be interesting. And we can look for what still > *could* be made parallel. I think it would be interesting to get the numbers for such a big machine for 3 cases: part1, part1+reuse timer calibration and part1+part2. Russ mentioned testing v6, so I guess the above numbers are for part1+reuse timer calibration. For my machine the smpboot times were: No patches: 700ms part 1:100ms part1+reuse timer calibration: 30ms part1+part2: 30ms Thanks, Usama
Doing INIT/SIPI/SIPI in parallel brings down the time for smpboot from ~700ms to 100ms (85% improvement) on a server with 128 CPUs split across 2 NUMA nodes. The parallel CPU bringup is disabled for all AMD CPUs in this version: (see discussions: https://lore.kernel.org/all/bc3f2b1332c4bb77558df8aa36493a55542fe5b9.camel@infradead.org/ and https://lore.kernel.org/all/3b6ac86fdc800cac5806433daf14a9095be101e9.camel@infradead.org/). The last patch reuses the timer calibration from CPU 0 for secondary CPUs which brings down the time for parallel smpboot from 100ms to 30ms. It is missing a sign-off from the author, which hopefully Arjan will add. Changes across versions: v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more v3: Clean up x2apic patch, add MTRR optimisation, lock topology update in preparation for more parallelisation. v4: Fixes to the real mode parallelisation patch spotted by SeanC, to avoid scribbling on initial_gs in common_cpu_up(), and to allow all 24 bits of the physical X2APIC ID to be used. That patch still needs a Signed-off-by from its original author, who once claimed not to remember writing it at all. But now we've fixed it, hopefully he'll admit it now :) v5: rebase to v6.1 and remeasure performance, disable parallel bringup for AMD CPUs. v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and reused timer calibration for secondary CPUs. Arjan van de Ven (1): x86/smpboot: reuse timer calibration David Woodhouse (9): x86/apic/x2apic: Fix parallel handling of cluster_mask cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() x86/smpboot: Split up native_cpu_up into separate phases and document them x86/smpboot: Disable parallel boot for AMD CPUs x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup x86/smpboot: Serialize topology updates for secondary bringup Thomas Gleixner (1): x86/smpboot: Support parallel startup of secondary CPUs arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/smp.h | 13 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/acpi/sleep.c | 1 + arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++---- arch/x86/kernel/cpu/amd.c | 11 + arch/x86/kernel/cpu/common.c | 6 +- arch/x86/kernel/cpu/mtrr/mtrr.c | 9 + arch/x86/kernel/head_64.S | 75 ++++++ arch/x86/kernel/smpboot.c | 333 ++++++++++++++++++-------- arch/x86/kernel/tsc.c | 3 + arch/x86/realmode/init.c | 3 + arch/x86/realmode/rm/trampoline_64.S | 14 ++ arch/x86/xen/smp_pv.c | 4 +- include/linux/cpuhotplug.h | 2 + include/linux/smpboot.h | 7 + kernel/cpu.c | 27 ++- kernel/smpboot.c | 2 +- kernel/smpboot.h | 2 - 21 files changed, 469 insertions(+), 159 deletions(-)