| Message ID | 20220905212458.1549179-1-vladimir.oltean@nxp.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [devicetree] arm64: dts: ls1028a-rdb: add more ethernet aliases | expand |
Am 2022-09-05 23:24, schrieb Vladimir Oltean: > Commit "arm64: dts: ls1028a: enable swp5 and eno3 for all boards" which > Shawn declared as applied, but for which I can't find a sha1sum, has > enabled a new Ethernet port on the LS1028A-RDB (&enetc_port3), but > U-Boot, which passes a MAC address to Linux' device tree through the > /aliases node, fails to do this for this newly enabled port. > > Fix that by adding more ethernet aliases in the only > backwards-compatible way possible: at the end of the current list. > > And since it is possible to very easily convert either swp4 or swp5 to > DSA user ports now (which have a MAC address of their own), using these > U-Boot commands: > > => fdt addr $fdt_addr_r > => fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@4 ethernet > > it would be good if those DSA user ports (swp4, swp5) gained a valid > MAC > address from U-Boot as well. In order for that to work properly, > provision two more ethernet aliases for &mscc_felix_port{4,5} as well. First, let me say, I'm fine with this patch. But I'm not sure, how many MAC addresses are actually reserved on your RDB/QDS boards? I guess, they being evaluation boards you don't care? ;) On the Kontron sl28 boards we reserve just 8 and that is already a lot for a board with max 6 out facing ports. 4 of these ports used to be a switch, so in theory it should work with 3 MAC addresses, right? Or even just 2 if there is no need to terminate any traffic on the switch interfaces. Anyway, do we really need so many addresses? What are the configurations here? For what is the address of the internal ports used? Let's say we are in the "port extender mode" and use the second internal port as an actual switch port, that would then be: 2x external enetc 1x internal enetc 4x external switch ports in port extender mode Which makes 7 addresses. The internal enetc port doesn't really make sense in a port extender mode, because there is no switching going on. So uhm, 6 addresses are the maximum? This is the MAC address distribution for now on the sl28 boards: https://lore.kernel.org/linux-devicetree/20220901221857.2600340-19-michael@walle.cc/ Please tell me if I'm missing something here. -michael > The resulting ordering is slightly unusual, but to me looks more > natural > than eno0, eno2, swp0, swp1, swp2, swp3, eno3, swp4, swp5. > > Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
On Tue, Sep 06, 2022 at 12:17:29AM +0200, Michael Walle wrote: > First, let me say, I'm fine with this patch. But I'm not sure, > how many MAC addresses are actually reserved on your > RDB/QDS boards? AFAIK, the Reference Design Boards are sold with an unprogrammed I2C EEPROM, but with a sticker containing 5 MAC addresses on the bottom of the board. It doesn't have a clear correspondence between MAC addresses and their intended use, although I suspect that one MAC address is intended for each RJ45 port (although that isn't how I use them). For the QIXIS Development Boards, I have no clue, it's probably even nonsensical to talk about MAC address reservations since there is just one onboard Ethernet port (RGMII) and the rest is routed via SERDES to PCIe slots, to pluggable riser cards, from which Linux/U-Boot don't bother too much to read back any info, even though I can't exclude something like an EEPROM may be available on those cards too. In any case, I think QDS boards don't leave the lab, so it doesn't matter too much. The way I use the MAC addresses from the sticker of my RDBs, on a day to day basis, is: ethaddr (eno0) - #1 eth1addr (eno2) - #2 eth2addr (swp0) - #2 eth3addr (swp1) - #2 eth4addr (swp2) - #2 eth5addr (swp3) - #2 And now I'm adding these new env variables: eth6addr (swp4) - #2 eth7addr (swp5) - #2 eth8addr (eno3) - #3 So I still have 2 more unique MAC addresses to burn through. > I guess, they being evaluation boards you don't care? ;) I do care a bit, but not that much. > On the Kontron sl28 boards we reserve just 8 and that is > already a lot for a board with max 6 out facing ports. 4 of > these ports used to be a switch, so in theory it should work /used/ to be a switch? What happened to them? Details? Or you mean "4 ports are used as a switch"? > with 3 MAC addresses, right? Which 3 MAC addresses would those be? Not sure I'm following. enetc #0, enetc #1, enetc #2? That could work, multiple DSA user ports can share the same MAC address (inherited from the DSA master or not) and things would work just fine unless you connect them to each other. > Or even just 2 if there is no need to terminate any traffic on the > switch interfaces. And here, which 2? enetc #0 and enetc #1? > Anyway, do we really need so many addresses? idk, who's "we" and what does "need" mean? (serious questions) I'm not sure I can give you any answer to this question. As an engineer working with the kernel, I need to roll the LS1028A Ethernet around on all its sides. The Linux RDB/QDS support will inevitably reflect what we need to test. Everybody else will have a fixed configuration, and the user reviews will vary from 'internet works! 5 stars!' to 'internet doesn't work! 1 star!'. To offer that quality of service for all front-facing ports, you don't need much. I know of a 12 port industrial switch that entered production with 1 MAC address, the "termination" address. It's fine, when it's marketed as a switch, people come to expect that and don't wonder too much. > What are the configurations here? For what is the address of the > internal ports used? By internal ports you mean swp4/swp5, or eno2/eno3? If eno2/eno3, then a configuration where having MAC addresses on these interfaces is useful to me is running some of the kselftests on the LS1028A-RDB, which does not have enough external enetc ports for 2 loopback pairs, so I do this, thereby having 1 external loopback through a cable, and 1 internal loopback in the SoC: ./psfp.sh eno0 swp0 swp4 eno2 https://github.com/torvalds/linux/blob/master/tools/testing/selftests/drivers/net/ocelot/psfp.sh Speaking of kselftests, it actually doesn't matter that much what the MAC addresses *are*, since we don't enter any network, just loop back traffic. In fact we have an environment variable STABLE_MAC_ADDRS, which when set, configures the ports to use some predetermined MAC addresses. There are other configurations where it is useful for eno2 to see DSA untagged traffic. These are downstream 802.1CB (where this hardware can offload redundant streams in the forwarding plane, but not in the termination plane, so we use eno2 as forwarding plane, for termination), DPDK on eno2 (which mainline Linux doesn't care about), and vfio-pci + QEMU, where DSA switch control still belongs to the Linux host, but the guest has 'internet'. > Let's say we are in the "port extender mode" and use the > second internal port as an actual switch port, that would > then be: > 2x external enetc > 1x internal enetc > 4x external switch ports in port extender mode > > Which makes 7 addresses. The internal enetc port doesn't > really make sense in a port extender mode, because there > is no switching going on. It can make sense. You can run ptp4l -i eno2, and ptp4l -i swp4, as separate processes, and you can get high quality synchronization between /dev/ptp0 (enetc) and /dev/ptp1 (felix) over internal Ethernet (there isn't any other mechanism in the SoC to keep them in sync if that is needed for some use case like a boundary_clock_jbod between eno0 + eno1 + swp0-swp3). > So uhm, 6 addresses are the maximum? No, the maximum is given by the number of ports, PFs and VFs. But that's a high number. It's the theoretical maximum. Then there's the practical maximum, which is given by what kind of embedded system is built with it. I think that the more general-purpose the system is, the more garden variety the networking use cases will be. I also think it would be very absurd for everybody to reserve a number of MAC addresses equal to the number of possibilities in which the LS1028A can expose IP termination points, if they're likely to never need them. > This is the MAC address distribution for now on the > sl28 boards: > https://lore.kernel.org/linux-devicetree/20220901221857.2600340-19-michael@walle.cc/ > > Please tell me if I'm missing something here. My 2 cents, if you don't need anything special like in-SoC PTP, 802.1CB, virtualization, and don't habitually connect ports of the same ports to each other or do some other sorts of redundant networking without VLANs, then there isn't too much wrong with one MAC address per RJ45 port, but best discuss with those who are actually marketing the devices.
Am 2022-09-06 01:54, schrieb Vladimir Oltean: > On Tue, Sep 06, 2022 at 12:17:29AM +0200, Michael Walle wrote: >> First, let me say, I'm fine with this patch. But I'm not sure, >> how many MAC addresses are actually reserved on your >> RDB/QDS boards? > > AFAIK, the Reference Design Boards are sold with an unprogrammed I2C > EEPROM, but with a sticker containing 5 MAC addresses on the bottom of > the board. It doesn't have a clear correspondence between MAC addresses > and their intended use, although I suspect that one MAC address is > intended for each RJ45 port (although that isn't how I use them). > > For the QIXIS Development Boards, I have no clue, it's probably even > nonsensical to talk about MAC address reservations since there is just > one onboard Ethernet port (RGMII) and the rest is routed via SERDES to > PCIe slots, to pluggable riser cards, from which Linux/U-Boot don't > bother > too much to read back any info, even though I can't exclude something > like an EEPROM may be available on those cards too. In any case, I > think > QDS boards don't leave the lab, so it doesn't matter too much. > > The way I use the MAC addresses from the sticker of my RDBs, on a day > to > day basis, is: > > ethaddr (eno0) - #1 > eth1addr (eno2) - #2 > eth2addr (swp0) - #2 > eth3addr (swp1) - #2 > eth4addr (swp2) - #2 > eth5addr (swp3) - #2 Ah, I never thought of handing out the same MAC address. > And now I'm adding these new env variables: > > eth6addr (swp4) - #2 > eth7addr (swp5) - #2 > eth8addr (eno3) - #3 > > So I still have 2 more unique MAC addresses to burn through. > >> I guess, they being evaluation boards you don't care? ;) > > I do care a bit, but not that much. > >> On the Kontron sl28 boards we reserve just 8 and that is >> already a lot for a board with max 6 out facing ports. 4 of >> these ports used to be a switch, so in theory it should work > > /used/ to be a switch? What happened to them? Details? Or you mean > "4 ports are used as a switch"? I shouldn't probably write mails right before going to sleep. Yes it should read "the 4 ports (swp0..swp3) are usually configured to as a switch." >> with 3 MAC addresses, right? > > Which 3 MAC addresses would those be? Not sure I'm following. enetc #0, > enetc #1, enetc #2? That could work, multiple DSA user ports can share > the same MAC address (inherited from the DSA master or not) and things > would work just fine unless you connect them to each other. enetc #0, #1 and swp0. As you mentioned, swp1..3 should inherit the address from swp0 then if swp0 is added as the first device, right? So why would enetc#2 (or #3) need a non-random mac address? I must be missing something here. >> Or even just 2 if there is no need to terminate any traffic on the >> switch interfaces. > > And here, which 2? enetc #0 and enetc #1? Yes. The switch would just be a dumb ethernet switch. >> Anyway, do we really need so many addresses? > > idk, who's "we" and what does "need" mean? (serious questions) We as in the users of the ls1028a SoC. And as I said, I thought of *unique* MAC addresses. > I'm not sure I can give you any answer to this question. As an engineer > working with the kernel, I need to roll the LS1028A Ethernet around on > all its sides. The Linux RDB/QDS support will inevitably reflect what > we > need to test. Everybody else will have a fixed configuration, and the > user reviews will vary from 'internet works! 5 stars!' to 'internet > doesn't work! 1 star!'. > > To offer that quality of service for all front-facing ports, you don't > need much. I know of a 12 port industrial switch that entered > production > with 1 MAC address, the "termination" address. It's fine, when it's > marketed as a switch, people come to expect that and don't wonder too > much. > >> What are the configurations here? For what is the address of the >> internal ports used? > > By internal ports you mean swp4/swp5, or eno2/eno3? eno2/eno3. > If eno2/eno3, then a > configuration where having MAC addresses on these interfaces is useful > to me is running some of the kselftests on the LS1028A-RDB, which does > not have enough external enetc ports for 2 loopback pairs, so I do > this, thereby having 1 external loopback through a cable, and 1 > internal > loopback in the SoC: > > ./psfp.sh eno0 swp0 swp4 eno2 > https://github.com/torvalds/linux/blob/master/tools/testing/selftests/drivers/net/ocelot/psfp.sh > > Speaking of kselftests, it actually doesn't matter that much what the > MAC addresses *are*, since we don't enter any network, just loop back > traffic. In fact we have an environment variable STABLE_MAC_ADDRS, > which > when set, configures the ports to use some predetermined MAC addresses. > > There are other configurations where it is useful for eno2 to see DSA > untagged traffic. These are downstream 802.1CB (where this hardware can > offload redundant streams in the forwarding plane, but not in the > termination plane, so we use eno2 as forwarding plane, for > termination), I'm not that familiar with 802.1CB. Is this MAC address visible outside of the switch or can it be a random one? > DPDK on eno2 (which mainline Linux doesn't care about), and vfio-pci + > QEMU, where DSA switch control still belongs to the Linux host, but the > guest has 'internet'. For me, all of that is kind of a trade off. If you want to use virtual interfaces, you might need to borrow a MAC address from one of the switch ports (where you have 3 unique addresses left if you combine all 4 ports to one bridge). >> Let's say we are in the "port extender mode" and use the >> second internal port as an actual switch port, that would >> then be: >> 2x external enetc >> 1x internal enetc >> 4x external switch ports in port extender mode >> >> Which makes 7 addresses. The internal enetc port doesn't >> really make sense in a port extender mode, because there >> is no switching going on. > > It can make sense. You can run ptp4l -i eno2, and ptp4l -i swp4, as > separate processes, and you can get high quality synchronization > between > /dev/ptp0 (enetc) and /dev/ptp1 (felix) over internal Ethernet (there > isn't any other mechanism in the SoC to keep them in sync if that is > needed for some use case like a boundary_clock_jbod between eno0 + eno1 > + swp0-swp3). Ok, could make sense. >> So uhm, 6 addresses are the maximum? > > No, the maximum is given by the number of ports, PFs and VFs. But > that's > a high number. It's the theoretical maximum. Then there's the practical > maximum, which is given by what kind of embedded system is built with > it. > I think that the more general-purpose the system is, the more garden > variety the networking use cases will be. I also think it would be very > absurd for everybody to reserve a number of MAC addresses equal to the > number of possibilities in which the LS1028A can expose IP termination > points, if they're likely to never need them. I think we are on the same track here. I was ignoring any VFs for now. So I guess, what I'm still missing here is why enet#2 and enet#3 (or even swp4 and swp5) would need a non-random MAC address. Except from your example above. Considering the usecase where swp0..3 is one bridge with eno2 and eno3 being the CPU ports. Then I'd only need a unique MAC address for eno0, eno1 and swp0, correct? -michael >> This is the MAC address distribution for now on the >> sl28 boards: >> https://lore.kernel.org/linux-devicetree/20220901221857.2600340-19-michael@walle.cc/ >> >> Please tell me if I'm missing something here. > > My 2 cents, if you don't need anything special like in-SoC PTP, > 802.1CB, > virtualization, and don't habitually connect ports of the same ports to > each other or do some other sorts of redundant networking without > VLANs, > then there isn't too much wrong with one MAC address per RJ45 port, but > best discuss with those who are actually marketing the devices.
On Tue, Sep 06, 2022 at 10:10:50AM +0200, Michael Walle wrote: > > > with 3 MAC addresses, right? > > > > Which 3 MAC addresses would those be? Not sure I'm following. enetc #0, > > enetc #1, enetc #2? That could work, multiple DSA user ports can share > > the same MAC address (inherited from the DSA master or not) and things > > would work just fine unless you connect them to each other. > > enetc #0, #1 and swp0. As you mentioned, swp1..3 should inherit the > address from swp0 then if swp0 is added as the first device, right? > > So why would enetc#2 (or #3) need a non-random mac address? I must > be missing something here. I didn't say that swp1..3 inherit the MAC address from swp0. I said that the DSA user ports can inherit the MAC address from the DSA master (eno2 or eno3). See dsa_slave_create(): if (!is_zero_ether_addr(port->mac)) // device tree eth_hw_addr_set(slave_dev, port->mac); else eth_hw_addr_inherit(slave_dev, master); The DSA user ports (swp0-swp3) can also share the same MAC address which is not inherited from the DSA master (eno2), but this can only be achieved through static configuration (such as manually setting the same mac-address in the device tree). > > If eno2/eno3, then a > > configuration where having MAC addresses on these interfaces is useful > > to me is running some of the kselftests on the LS1028A-RDB, which does > > not have enough external enetc ports for 2 loopback pairs, so I do > > this, thereby having 1 external loopback through a cable, and 1 internal > > loopback in the SoC: > > > > ./psfp.sh eno0 swp0 swp4 eno2 > > https://github.com/torvalds/linux/blob/master/tools/testing/selftests/drivers/net/ocelot/psfp.sh > > > > Speaking of kselftests, it actually doesn't matter that much what the > > MAC addresses *are*, since we don't enter any network, just loop back > > traffic. In fact we have an environment variable STABLE_MAC_ADDRS, which > > when set, configures the ports to use some predetermined MAC addresses. > > > > There are other configurations where it is useful for eno2 to see DSA > > untagged traffic. These are downstream 802.1CB (where this hardware can > > offload redundant streams in the forwarding plane, but not in the > > termination plane, so we use eno2 as forwarding plane, for termination), > > I'm not that familiar with 802.1CB. Is this MAC address visible outside > of the switch or can it be a random one? The MAC address of eno2 is visible outside the switch, onto the network. Take this 802.1CB ring network for example: https://github.com/vladimiroltean/isochron/blob/master/frer/8021cb.sh Every board has 2 redundant ports in the ring, and every board can ping every other board through the redundant ring, through an IP termination point (eno2). Except that the switch does not support inserting/stripping 802.1CB redundancy headers for locally terminated traffic, or splitting/ eliminating duplicate streams for locally terminated traffic. So we configure the switch to consider the internal swp4 as a user port, and effectively set it up for the forwarding plane, in a bridge with swp0 and swp1. It's now as if each eno2 used for termination is connected to a switch that's physically separate from the LS1028A, which handles the redundant forwarding. Except it's not physically separate, it's part of the same SoC, and its DSA master is eno3. All in all, it requires a bit of a split brain syndrome to work with it. This is in fact one of the key things blocking 802.1CB upstreaming for us, as a side note - we don't have a very good proposal for modeling a generic software data path for this. You can see more details here if you care enough: https://patchwork.kernel.org/project/netdevbpf/patch/20210928114451.24956-1-xiaoliang.yang_1@nxp.com/ > > DPDK on eno2 (which mainline Linux doesn't care about), and vfio-pci + > > QEMU, where DSA switch control still belongs to the Linux host, but the > > guest has 'internet'. > > For me, all of that is kind of a trade off. If you want to use > virtual interfaces, you might need to borrow a MAC address from > one of the switch ports (where you have 3 unique addresses left > if you combine all 4 ports to one bridge). I didn't say virtual interfaces in this case, I said vfio-pci which is a way to do PCI direct assignment to a guest OS, which sees the whole 0000:00:00.2 PF corresponding to eno2. By the way, the internal enetc ports eno2 and eno3 don't have VFs since SR-IOV is a bit of a complicated topic when DSA-tagged traffic is expected (you don't want a guest OS to see DSA tags in packets, it doesn't know what to do with them). But this is one of the main reasons we have 2 internal ports btw. You configure eno3 as a DSA master, you leave swp4 as a user port, so now eno2 sees DSA untagged traffic and you can do direct assignment to whomever. > > > So uhm, 6 addresses are the maximum? > > > > No, the maximum is given by the number of ports, PFs and VFs. But that's > > a high number. It's the theoretical maximum. Then there's the practical > > maximum, which is given by what kind of embedded system is built with > > it. > > I think that the more general-purpose the system is, the more garden > > variety the networking use cases will be. I also think it would be very > > absurd for everybody to reserve a number of MAC addresses equal to the > > number of possibilities in which the LS1028A can expose IP termination > > points, if they're likely to never need them. > > I think we are on the same track here. I was ignoring any VFs for now. > So I guess, what I'm still missing here is why enet#2 and enet#3 (or > even swp4 and swp5) would need a non-random MAC address. Except from > your example above. Considering the usecase where swp0..3 is one bridge > with eno2 and eno3 being the CPU ports. Then I'd only need a unique > MAC address for eno0, eno1 and swp0, correct? Don't say "unique MAC address for swp0", since swp0's MAC address is not unique, you probably mean to say "a MAC address which will be shared by swp0-swp3". I think I've answered why eno2/eno3 could need a stable MAC address - for the case when they aren't used as DSA masters (through the switch termination plane) but as interfaces unaware of the switch connected to them (through the switch forwarding plane).
Am 2022-09-06 12:05, schrieb Vladimir Oltean: > On Tue, Sep 06, 2022 at 10:10:50AM +0200, Michael Walle wrote: >> > > with 3 MAC addresses, right? >> > >> > Which 3 MAC addresses would those be? Not sure I'm following. enetc #0, >> > enetc #1, enetc #2? That could work, multiple DSA user ports can share >> > the same MAC address (inherited from the DSA master or not) and things >> > would work just fine unless you connect them to each other. >> >> enetc #0, #1 and swp0. As you mentioned, swp1..3 should inherit the >> address from swp0 then if swp0 is added as the first device, right? >> >> So why would enetc#2 (or #3) need a non-random mac address? I must >> be missing something here. > > I didn't say that swp1..3 inherit the MAC address from swp0. I said > that > the DSA user ports can inherit the MAC address from the DSA master > (eno2 > or eno3). See dsa_slave_create(): > > if (!is_zero_ether_addr(port->mac)) // device tree > eth_hw_addr_set(slave_dev, port->mac); > else > eth_hw_addr_inherit(slave_dev, master); > > The DSA user ports (swp0-swp3) can also share the same MAC address > which > is not inherited from the DSA master (eno2), but this can only be > achieved through static configuration (such as manually setting the > same > mac-address in the device tree). Right. I need to jog my memory on the whole briding stuff again. Sorry, I mixed that up with the bridge using the lowest MAC as its id. I.e. it looked like br0 inherited the address from the interface which was added to it first. >> > If eno2/eno3, then a >> > configuration where having MAC addresses on these interfaces is useful >> > to me is running some of the kselftests on the LS1028A-RDB, which does >> > not have enough external enetc ports for 2 loopback pairs, so I do >> > this, thereby having 1 external loopback through a cable, and 1 internal >> > loopback in the SoC: >> > >> > ./psfp.sh eno0 swp0 swp4 eno2 >> > https://github.com/torvalds/linux/blob/master/tools/testing/selftests/drivers/net/ocelot/psfp.sh >> > >> > Speaking of kselftests, it actually doesn't matter that much what the >> > MAC addresses *are*, since we don't enter any network, just loop back >> > traffic. In fact we have an environment variable STABLE_MAC_ADDRS, which >> > when set, configures the ports to use some predetermined MAC addresses. >> > >> > There are other configurations where it is useful for eno2 to see DSA >> > untagged traffic. These are downstream 802.1CB (where this hardware can >> > offload redundant streams in the forwarding plane, but not in the >> > termination plane, so we use eno2 as forwarding plane, for termination), >> >> I'm not that familiar with 802.1CB. Is this MAC address visible >> outside >> of the switch or can it be a random one? > > The MAC address of eno2 is visible outside the switch, onto the > network. > Take this 802.1CB ring network for example: > https://github.com/vladimiroltean/isochron/blob/master/frer/8021cb.sh > > Every board has 2 redundant ports in the ring, and every board can ping > every other board through the redundant ring, through an IP termination > point (eno2). Except that the switch does not support > inserting/stripping > 802.1CB redundancy headers for locally terminated traffic, or > splitting/ > eliminating duplicate streams for locally terminated traffic. > > So we configure the switch to consider the internal swp4 as a user > port, > and effectively set it up for the forwarding plane, in a bridge with > swp0 and swp1. It's now as if each eno2 used for termination is > connected to a switch that's physically separate from the LS1028A, > which > handles the redundant forwarding. Except it's not physically separate, > it's part of the same SoC, and its DSA master is eno3. All in all, it > requires a bit of a split brain syndrome to work with it. This is in > fact one of the key things blocking 802.1CB upstreaming for us, as a > side note - we don't have a very good proposal for modeling a generic > software data path for this. > > You can see more details here if you care enough: > https://patchwork.kernel.org/project/netdevbpf/patch/20210928114451.24956-1-xiaoliang.yang_1@nxp.com/ > >> > DPDK on eno2 (which mainline Linux doesn't care about), and vfio-pci + >> > QEMU, where DSA switch control still belongs to the Linux host, but the >> > guest has 'internet'. >> >> For me, all of that is kind of a trade off. If you want to use >> virtual interfaces, you might need to borrow a MAC address from >> one of the switch ports (where you have 3 unique addresses left >> if you combine all 4 ports to one bridge). > > I didn't say virtual interfaces in this case, I said vfio-pci which is > a > way to do PCI direct assignment to a guest OS, which sees the whole > 0000:00:00.2 PF corresponding to eno2. Yes of course. I need to find a good compromise for the MAC assignments. And running virtualization on our board might be a use case, but I don't know how common that one is. So for now, I'm ignoring it. > By the way, the internal enetc ports eno2 and eno3 don't have VFs since > SR-IOV is a bit of a complicated topic when DSA-tagged traffic is > expected (you don't want a guest OS to see DSA tags in packets, it > doesn't know what to do with them). But this is one of the main reasons > we have 2 internal ports btw. You configure eno3 as a DSA master, you > leave swp4 as a user port, so now eno2 sees DSA untagged traffic and > you > can do direct assignment to whomever. > >> > > So uhm, 6 addresses are the maximum? >> > >> > No, the maximum is given by the number of ports, PFs and VFs. But that's >> > a high number. It's the theoretical maximum. Then there's the practical >> > maximum, which is given by what kind of embedded system is built with >> > it. >> > I think that the more general-purpose the system is, the more garden >> > variety the networking use cases will be. I also think it would be very >> > absurd for everybody to reserve a number of MAC addresses equal to the >> > number of possibilities in which the LS1028A can expose IP termination >> > points, if they're likely to never need them. >> >> I think we are on the same track here. I was ignoring any VFs for now. >> So I guess, what I'm still missing here is why enet#2 and enet#3 (or >> even swp4 and swp5) would need a non-random MAC address. Except from >> your example above. Considering the usecase where swp0..3 is one >> bridge >> with eno2 and eno3 being the CPU ports. Then I'd only need a unique >> MAC address for eno0, eno1 and swp0, correct? > > Don't say "unique MAC address for swp0", since swp0's MAC address is > not > unique, you probably mean to say "a MAC address which will be shared by > swp0-swp3". That I actually don't understand. I have the following addresses after booting: # ip link .. 4: gbe0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:a0:a5:5c:6b:62 brd ff:ff:ff:ff:ff:ff inet 172.16.1.2/24 scope global gbe0 valid_lft forever preferred_lft forever inet6 fe80::2a0:a5ff:fe5c:6b62/64 scope link valid_lft forever preferred_lft forever 5: gbe1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:a0:a5:5c:6b:63 brd ff:ff:ff:ff:ff:ff 6: eno2: <BROADCAST,MULTICAST> mtu 1504 qdisc noop state DOWN group default qlen 1000 link/ether 8e:6c:20:8a:ab:52 brd ff:ff:ff:ff:ff:ff 7: eno3: <BROADCAST,MULTICAST> mtu 1504 qdisc noop state DOWN group default qlen 1000 link/ether c6:fd:b1:88:3c:36 brd ff:ff:ff:ff:ff:ff 8: swp0@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:a0:a5:5c:6b:66 brd ff:ff:ff:ff:ff:ff 9: swp1@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:a0:a5:5c:6b:67 brd ff:ff:ff:ff:ff:ff 10: swp2@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:a0:a5:5c:6b:68 brd ff:ff:ff:ff:ff:ff 11: swp3@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:a0:a5:5c:6b:69 brd ff:ff:ff:ff:ff:ff gbe0 is eno0 and gbe1 is eno1. So in my case they are unique. When adding all the ports to a bridge, the bridge gets the lowest MAC. # ip link add name br0 type bridge # ip link set swp0 master br0 # ip link set swp1 master br0 # ip link set swp2 master br0 # ip link set swp3 master br0 12: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 00:a0:a5:5c:6b:66 brd ff:ff:ff:ff:ff:ff Is that what you mean with "a MAC address which will be shared by swp0-swp3"? > I think I've answered why eno2/eno3 could need a stable MAC address - > for the case when they aren't used as DSA masters (through the switch > termination plane) but as interfaces unaware of the switch connected to > them (through the switch forwarding plane). Yes thanks for the explanation! -michael
On Wed, Sep 07, 2022 at 10:56:37AM +0200, Michael Walle wrote: > > > I think we are on the same track here. I was ignoring any VFs for now. > > > So I guess, what I'm still missing here is why enet#2 and enet#3 (or > > > even swp4 and swp5) would need a non-random MAC address. Except from > > > your example above. Considering the usecase where swp0..3 is one bridge > > > with eno2 and eno3 being the CPU ports. Then I'd only need a unique > > > MAC address for eno0, eno1 and swp0, correct? > > > > Don't say "unique MAC address for swp0", since swp0's MAC address is not > > unique, you probably mean to say "a MAC address which will be shared by > > swp0-swp3". > > That I actually don't understand. I have the following addresses after > booting: > > # ip link > .. > 4: gbe0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group > default qlen 1000 > link/ether 00:a0:a5:5c:6b:62 brd ff:ff:ff:ff:ff:ff > inet 172.16.1.2/24 scope global gbe0 > valid_lft forever preferred_lft forever > inet6 fe80::2a0:a5ff:fe5c:6b62/64 scope link > valid_lft forever preferred_lft forever > 5: gbe1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default > qlen 1000 > link/ether 00:a0:a5:5c:6b:63 brd ff:ff:ff:ff:ff:ff > 6: eno2: <BROADCAST,MULTICAST> mtu 1504 qdisc noop state DOWN group default > qlen 1000 > link/ether 8e:6c:20:8a:ab:52 brd ff:ff:ff:ff:ff:ff > 7: eno3: <BROADCAST,MULTICAST> mtu 1504 qdisc noop state DOWN group default > qlen 1000 > link/ether c6:fd:b1:88:3c:36 brd ff:ff:ff:ff:ff:ff > 8: swp0@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN > group default qlen 1000 > link/ether 00:a0:a5:5c:6b:66 brd ff:ff:ff:ff:ff:ff > 9: swp1@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN > group default qlen 1000 > link/ether 00:a0:a5:5c:6b:67 brd ff:ff:ff:ff:ff:ff > 10: swp2@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN > group default qlen 1000 > link/ether 00:a0:a5:5c:6b:68 brd ff:ff:ff:ff:ff:ff > 11: swp3@eno2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN > group default qlen 1000 > link/ether 00:a0:a5:5c:6b:69 brd ff:ff:ff:ff:ff:ff > > gbe0 is eno0 and gbe1 is eno1. So in my case they are unique. > > When adding all the ports to a bridge, the bridge gets the lowest MAC. > > # ip link add name br0 type bridge > # ip link set swp0 master br0 > # ip link set swp1 master br0 > # ip link set swp2 master br0 > # ip link set swp3 master br0 > > 12: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT > group default qlen 1000 > link/ether 00:a0:a5:5c:6b:66 brd ff:ff:ff:ff:ff:ff > > Is that what you mean with "a MAC address which will be shared by > swp0-swp3"? No, what I'm trying is only to reformulate what you've said, "Then I'd only need a unique MAC address for eno0, eno1 and swp0". My understanding of what you mean by "unique address for swp0" is that the addresses for swp1-swp3 don't matter (hence they can be set to the same address as swp0) since there will be a bridge that serves as the IP termination point, with its own MAC address, inherited from the first bridge port, swp0. But this makes it improper to call it a "unique address for swp0".
On Tue, Sep 06, 2022 at 12:24:58AM +0300, Vladimir Oltean wrote: > Commit "arm64: dts: ls1028a: enable swp5 and eno3 for all boards" which > Shawn declared as applied, but for which I can't find a sha1sum, has > enabled a new Ethernet port on the LS1028A-RDB (&enetc_port3), but > U-Boot, which passes a MAC address to Linux' device tree through the > /aliases node, fails to do this for this newly enabled port. > > Fix that by adding more ethernet aliases in the only > backwards-compatible way possible: at the end of the current list. > > And since it is possible to very easily convert either swp4 or swp5 to > DSA user ports now (which have a MAC address of their own), using these > U-Boot commands: > > => fdt addr $fdt_addr_r > => fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@4 ethernet > > it would be good if those DSA user ports (swp4, swp5) gained a valid MAC > address from U-Boot as well. In order for that to work properly, > provision two more ethernet aliases for &mscc_felix_port{4,5} as well. > > The resulting ordering is slightly unusual, but to me looks more natural > than eno0, eno2, swp0, swp1, swp2, swp3, eno3, swp4, swp5. > > Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com> Applied, thanks!
diff --git a/arch/arm64/boot/dts/freescale/fsl-ls1028a-rdb.dts b/arch/arm64/boot/dts/freescale/fsl-ls1028a-rdb.dts index e33725c60169..ecd2c1ea177f 100644 --- a/arch/arm64/boot/dts/freescale/fsl-ls1028a-rdb.dts +++ b/arch/arm64/boot/dts/freescale/fsl-ls1028a-rdb.dts @@ -29,6 +29,9 @@ aliases { ethernet3 = &mscc_felix_port1; ethernet4 = &mscc_felix_port2; ethernet5 = &mscc_felix_port3; + ethernet6 = &mscc_felix_port4; + ethernet7 = &mscc_felix_port5; + ethernet8 = &enetc_port3; }; chosen {
Commit "arm64: dts: ls1028a: enable swp5 and eno3 for all boards" which Shawn declared as applied, but for which I can't find a sha1sum, has enabled a new Ethernet port on the LS1028A-RDB (&enetc_port3), but U-Boot, which passes a MAC address to Linux' device tree through the /aliases node, fails to do this for this newly enabled port. Fix that by adding more ethernet aliases in the only backwards-compatible way possible: at the end of the current list. And since it is possible to very easily convert either swp4 or swp5 to DSA user ports now (which have a MAC address of their own), using these U-Boot commands: => fdt addr $fdt_addr_r => fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@4 ethernet it would be good if those DSA user ports (swp4, swp5) gained a valid MAC address from U-Boot as well. In order for that to work properly, provision two more ethernet aliases for &mscc_felix_port{4,5} as well. The resulting ordering is slightly unusual, but to me looks more natural than eno0, eno2, swp0, swp1, swp2, swp3, eno3, swp4, swp5. Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com> --- arch/arm64/boot/dts/freescale/fsl-ls1028a-rdb.dts | 3 +++ 1 file changed, 3 insertions(+)