| Message ID | 20210507085756.20427-1-yangbo.lu@nxp.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | ptp: support virtual clocks for multiple domains | expand |
On Fri, May 07, 2021 at 04:57:50PM +0800, Yangbo Lu wrote: > ptp4l -i eno0 -p/dev/ptp1 -m --domainNumber=1 --priority1=128 > domain1-slave.log 2>&1 & > ptp4l -i eno0 -p/dev/ptp2 -m --domainNumber=2 --priority1=128 > domain2-slave.log 2>&1 & > ptp4l -i eno0 -p/dev/ptp3 -m --domainNumber=3 --priority1=127 > domain3-master.log 2>&1 & > - Make changing on physical clock transparent to virtual clocks. > The virtual clock is based on free running physical clock. If physical > clock has to be changed, how to make the change transparent to all > virtual clocks? Yes, this is a serious defect of this patch series, and there is no way to fix it. In the above example, suppose that domainNumber 1 needs +11 ppm and domainNumber 2 needs -12 ppm. You can't adjust one clock in two different ways. > Actually the frequency adjustmend on physical clock > may be hidden by updating virtual clocks in opposite direction at same > time. Considering the ppb values adjusted, the code execution delay > will be little enough to ignore. Assuming that the frequency offset is exactly the same on all domains, which will very often be false. > But it's hard to hide clock stepping, > by now I think the workaround may be inhibiting physical clock stepping > when run user space ptp application. That won't work either, because a phase offset on one domain will result in a large slew at the maximum rate, but that rate would spoil the other domains. The best way to support multiple PTP domains simultaneously is in the application. It is really the only way, because the kernel does not handle any details of the PTP, like domainNumber. The kernel only provides clock control and packet time stamping. ptp4l does not handle multiple domains today, but it definitely could be added with some effort. It would have to synchronize the clock to one chosen domain, and track the phase and frequency offsets of each of the other domains with respect to the chosen domain. Having done this, the software can convert time stamps between the domains perfectly. Using the tracked phase and frequency offsets, it can also switch domains seamlessly without hacks or guesswork. So I have to say NAK to this series because it can't do any of that, and it cannot be made to work either. Thanks, Richard
Hi Richard, > -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: 2021年5月9日 3:17 > To: Y.b. Lu <yangbo.lu@nxp.com> > Cc: netdev@vger.kernel.org; David S . Miller <davem@davemloft.net>; Claudiu > Manoil <claudiu.manoil@nxp.com>; Jakub Kicinski <kuba@kernel.org> > Subject: Re: [net-next 0/6] ptp: support virtual clocks for multiple domains > > On Fri, May 07, 2021 at 04:57:50PM +0800, Yangbo Lu wrote: > > ptp4l -i eno0 -p/dev/ptp1 -m --domainNumber=1 --priority1=128 > > domain1-slave.log 2>&1 & > > ptp4l -i eno0 -p/dev/ptp2 -m --domainNumber=2 --priority1=128 > > domain2-slave.log 2>&1 & > > ptp4l -i eno0 -p/dev/ptp3 -m --domainNumber=3 --priority1=127 > > > domain3-master.log 2>&1 & > > > - Make changing on physical clock transparent to virtual clocks. > > The virtual clock is based on free running physical clock. If physical > > clock has to be changed, how to make the change transparent to all > > virtual clocks? > > Yes, this is a serious defect of this patch series, and there is no way to fix it. In > the above example, suppose that domainNumber 1 needs +11 ppm and > domainNumber 2 needs -12 ppm. You can't adjust one clock in two different > ways. There may be some misunderstanding. In the example, domain 1, 2, 3 are based on PTP virtual clocks ptp1, ptp2 and ptp3 which are utilizing their own timecounter. The clock adjustment on them won't affect each other. The example worked fine in my verification. > > > Actually the frequency adjustmend on physical clock > > may be hidden by updating virtual clocks in opposite direction at same > > time. Considering the ppb values adjusted, the code execution delay > > will be little enough to ignore. > > Assuming that the frequency offset is exactly the same on all domains, which > will very often be false. I mean if the physical clock keeps free running, all virtual clocks utilizing their own timercounter can work fine independently without affecting on each other. If the physical clock has change on frequency, there is also way to make the change not affect virtual clocks. For example, when there is +12 ppm change on physical clock, just give -12 ppm change on virtual clocks. And the code execution delay just has very little affecting on the time error considering only 12 ppm frequency adjusted. This is a TODO work. > > > But it's hard to hide clock stepping, > > by now I think the workaround may be inhibiting physical clock stepping > > when run user space ptp application. > > That won't work either, because a phase offset on one domain will result in a > large slew at the maximum rate, but that rate would spoil the other domains. Phase offset changing on physical clock affects virtual clocks. This is not able to address. But there is workaround, like we inhibit physical clock stepping during PTP application running. > > The best way to support multiple PTP domains simultaneously is in the > application. It is really the only way, because the kernel does not handle any > details of the PTP, like domainNumber. The kernel only provides clock control > and packet time stamping. I understand. I explain my idea above to make it clear avoid misunderstanding. This patch-set focuses on clock control and timestamping. It provides support for virtual clocks and domain timestamp related. Put these in kernel, PTP applications don't have to do such thing by each of them. > > ptp4l does not handle multiple domains today, but it definitely could be added > with some effort. It would have to synchronize the clock to one chosen > domain, and track the phase and frequency offsets of each of the other > domains with respect to the chosen domain. Having done this, the software > can convert time stamps between the domains perfectly. Using the tracked > phase and frequency offsets, it can also switch domains seamlessly without > hacks or guesswork. Sure. This can be done in application. I'm not sure whether my explaining address the "defect" you mentioned :) I hope yes. Actually this patch-set can provide ptp1, ptp2, ... , which are standard ptp devices with domains bound. ptp4l can directly use them for domain clock controls. The timestamp is already converted to domain in kernel. This benefits all PTP applications. > > So I have to say NAK to this series because it can't do any of that, and it cannot > be made to work either. There may be misunderstanding. Hope for comments after seeing my explaining:) Thank you. > > Thanks, > Richard
On Mon, May 10, 2021 at 03:04:39AM +0000, Y.b. Lu wrote: > There may be some misunderstanding. In the example, domain 1, 2, 3 are based on PTP virtual clocks ptp1, ptp2 and ptp3 which are utilizing their own timecounter. > The clock adjustment on them won't affect each other. The example worked fine in my verification. Okay, now I think I understand what you are suggesting. Still, there are issues you haven't considered. > I mean if the physical clock keeps free running, all virtual clocks utilizing their own timercounter can work fine independently without affecting on each other. Right. This is critical! > If the physical clock has change on frequency, there is also way to make the change not affect virtual clocks. > For example, when there is +12 ppm change on physical clock, just give -12 ppm change on virtual clocks. That will cause issues in very many cases. For example, what happens when the "real" clock sees a large offset, and it doesn't step, but rather applies the maximum frequency offset to slew the clock? That maximum might be larger that the max possible in the virtual clocks. Even with smaller frequency offsets, the un-synchronized, quasi random changes in the "real" clock will spoil the virtual clocks. I won't support such an approach. However, if the "real" clock is guaranteed to stay free running, and the virtual clocks give up any hope of producing periodic outputs, then your idea might work. Thinking out loud: You could make a sysfs knob that converts a "real" clock into two or more virtual clocks. For example: cat /sys/class/ptp/ptp0/number_vclocks 0 # ptp0 is a "real" clock echo 3 > /sys/class/ptp/ptp0/number_vclocks # This resets the frequency offset to zero and creates three # new clocks using timecounter numeric adjustment, ptp0, 1, and 2. # ptp0 loses its periodic output abilities. # ptp0 is now a virtual clock, just like ptp1 and 2. echo 0 > /sys/class/ptp/ptp0/number_vclocks # back to normal again. In addition to that, you will need a way to make the relationships between the clocks and the network interfaces discoverable. It needs more thought and careful design, but I think having clock_gettime() available for the different clocks would be nice to have for the applications. Thanks, Richard
Hi Richard, > -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: 2021年5月11日 7:19 > To: Y.b. Lu <yangbo.lu@nxp.com> > Cc: netdev@vger.kernel.org; David S . Miller <davem@davemloft.net>; Claudiu > Manoil <claudiu.manoil@nxp.com>; Jakub Kicinski <kuba@kernel.org> > Subject: Re: [net-next 0/6] ptp: support virtual clocks for multiple domains > > On Mon, May 10, 2021 at 03:04:39AM +0000, Y.b. Lu wrote: > > > There may be some misunderstanding. In the example, domain 1, 2, 3 are > based on PTP virtual clocks ptp1, ptp2 and ptp3 which are utilizing their own > timecounter. > > The clock adjustment on them won't affect each other. The example worked > fine in my verification. > > Okay, now I think I understand what you are suggesting. Still, there are issues > you haven't considered. > > > I mean if the physical clock keeps free running, all virtual clocks utilizing their > own timercounter can work fine independently without affecting on each > other. > > Right. This is critical! > > > If the physical clock has change on frequency, there is also way to make the > change not affect virtual clocks. > > For example, when there is +12 ppm change on physical clock, just give -12 > ppm change on virtual clocks. > > That will cause issues in very many cases. > > For example, what happens when the "real" clock sees a large offset, and it > doesn't step, but rather applies the maximum frequency offset to slew the > clock? That maximum might be larger that the max possible in the virtual > clocks. Even with smaller frequency offsets, the un-synchronized, quasi > random changes in the "real" clock will spoil the virtual clocks. I won't > support such an approach. > What I thought was in code writing registers to adjust physical clock frequency, and immediately adjusting virtual clocks in opposite direction. Make the operations atomic by locking. Assume the code execution has a DELAY, and the frequency adjusted is PPM. Then the time error affecting on virtual clock will be DELAY * PPM. I'm not sure what the DELAY value will be on other platforms. Just for example, for 1us delay, 1000ppm adjustment will have 1ns time error. But indeed, this approach may be not feasible as you said. Especially it is adjusting clock in max frequency, and there are many virtual clocks. The time error may be large enough to cause issues. (I'm not sure whether I understand you correctly, sorry.) So, a question is, for hardware which supports only one PTP clock, can multiple domains be supported where physical clock also participates in synchronization of a domain? (Because sometime the physical clock is required to be synchronized for TSN using, or other usages.) Do you think it's possible? > However, if the "real" clock is guaranteed to stay free running, and the virtual > clocks give up any hope of producing periodic outputs, then your idea might > work. > > Thinking out loud: You could make a sysfs knob that converts a "real" > clock into two or more virtual clocks. For example: > > cat /sys/class/ptp/ptp0/number_vclocks > 0 # ptp0 is a "real" clock > > echo 3 > /sys/class/ptp/ptp0/number_vclocks > # This resets the frequency offset to zero and creates three > # new clocks using timecounter numeric adjustment, ptp0, 1, and 2. > # ptp0 loses its periodic output abilities. > # ptp0 is now a virtual clock, just like ptp1 and 2. > > echo 0 > /sys/class/ptp/ptp0/number_vclocks > # back to normal again. > > In addition to that, you will need a way to make the relationships between the > clocks and the network interfaces discoverable. Agree. This should be done carefully and everything should be considered. Will converting physical clock ptp0 to virtual clock ptp0 introduce more effort to implement, comparing to keep physical clock ptp0 but limit to use it? > > It needs more thought and careful design, but I think having > clock_gettime() available for the different clocks would be nice to have for the > applications. Thank you. Then regarding the domain timestamp, do you think it's proper to do the conversion in kernel as I implemented. > > Thanks, > Richard
On Tue, May 11, 2021 at 10:40:15AM +0000, Y.b. Lu wrote: > What I thought was in code writing registers to adjust physical clock frequency, and immediately adjusting virtual clocks in opposite direction. > Make the operations atomic by locking. Assume the code execution has a DELAY, and the frequency adjusted is PPM. > Then the time error affecting on virtual clock will be DELAY * PPM. I'm not sure what the DELAY value will be on other platforms. > Just for example, for 1us delay, 1000ppm adjustment will have 1ns time error. > > But indeed, this approach may be not feasible as you said. Especially it is adjusting clock in max frequency, and there are many virtual clocks. > The time error may be large enough to cause issues. (I'm not sure whether I understand you correctly, sorry.) You understand correctly. > So, a question is, for hardware which supports only one PTP clock, can multiple domains be supported where physical clock also participates in synchronization of a domain? > (Because sometime the physical clock is required to be synchronized for TSN using, or other usages.) > Do you think it's possible? No, it won't work. You can't adjust both the physical clock and the timecounters at the same time. The code would be an awful hack, and it would not work in all real world circumstances. If the kernel offers a new time service, then it has to work always. So, getting back to my user space idea, it _would_ work to let the application stack control the HW clock as before, but to track the other domains numerically. Then, the other applications could use the TIME_STATUS_NP management message (designed for use with gPTP and free_running) to get the current time in the other domains. So take your pick. You can't have it both ways, I'm afraid. > > In addition to that, you will need a way to make the relationships between the > > clocks and the network interfaces discoverable. > > Agree. This should be done carefully and everything should be considered. > Will converting physical clock ptp0 to virtual clock ptp0 introduce more effort to implement, > comparing to keep physical clock ptp0 but limit to use it? I think either way, it would be a substantial change in the kernel code. > > It needs more thought and careful design, but I think having > > clock_gettime() available for the different clocks would be nice to have for the > > applications. > > Thank you. Then regarding the domain timestamp, do you think it's proper to do the conversion in kernel as I implemented. Yes, it would be very nice for the applications, because they wouldn't have to use a different API for gettime(). The drawback is that you loose the ability to generate synchronized signals in HW from the physical clock. (Time stamping inputs would still work, because the timecounter code cleverly allows that.) Thanks, Richard
Hi Richard, > -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: 2021年5月11日 23:50 > To: Y.b. Lu <yangbo.lu@nxp.com> > Cc: netdev@vger.kernel.org; David S . Miller <davem@davemloft.net>; Claudiu > Manoil <claudiu.manoil@nxp.com>; Jakub Kicinski <kuba@kernel.org> > Subject: Re: [net-next 0/6] ptp: support virtual clocks for multiple domains > > On Tue, May 11, 2021 at 10:40:15AM +0000, Y.b. Lu wrote: > > What I thought was in code writing registers to adjust physical clock > frequency, and immediately adjusting virtual clocks in opposite direction. > > Make the operations atomic by locking. Assume the code execution has a > DELAY, and the frequency adjusted is PPM. > > Then the time error affecting on virtual clock will be DELAY * PPM. I'm not > sure what the DELAY value will be on other platforms. > > Just for example, for 1us delay, 1000ppm adjustment will have 1ns time > error. > > > > But indeed, this approach may be not feasible as you said. Especially it is > adjusting clock in max frequency, and there are many virtual clocks. > > The time error may be large enough to cause issues. (I'm not sure > > whether I understand you correctly, sorry.) > > You understand correctly. > > > So, a question is, for hardware which supports only one PTP clock, can > multiple domains be supported where physical clock also participates in > synchronization of a domain? > > (Because sometime the physical clock is required to be synchronized > > for TSN using, or other usages.) Do you think it's possible? > > No, it won't work. You can't adjust both the physical clock and the > timecounters at the same time. The code would be an awful hack, and it > would not work in all real world circumstances. If the kernel offers a new > time service, then it has to work always. > > So, getting back to my user space idea, it _would_ work to let the application > stack control the HW clock as before, but to track the other domains > numerically. Then, the other applications could use the TIME_STATUS_NP > management message (designed for use with gPTP and > free_running) to get the current time in the other domains. > > So take your pick. You can't have it both ways, I'm afraid. > I give up supporting physical clock and the timecounters adjusting at the same time, but I may continue to support virtual clock per your suggestion. Getting back to your user space idea, I'd like to understand further to see if I can make some contribution. Actually I can't think out how to track (there is not timecouner like in kernel) in a easy way, and I have some concerns too. I assume we have a way for physical clock domain to track phase and frequency offset for each of other domains. Are the phase and frequency offset enough to convert timestamp? Still the key problem is hiding physical clock changes for the virtual. There is another reason that I initially gave workaround of no stepping on physical clock. Because timestamping is asynchronous with physical clock phase changing. When a timestamp is captured and physical clock time has a small change, we have no idea if timestamping happened before or after clock changing during conversion. Finally gPTP standard does use management messages. I think it doesn’t matter only if it implements the function we need. Maybe I haven't got your point... Thanks a lot.
On Fri, May 14, 2021 at 06:41:28AM +0000, Y.b. Lu wrote: > I give up supporting physical clock and the timecounters adjusting at the same time, but I may continue to support virtual clock per your suggestion. Okay, so the physical clock stays free running when virtual clocks are active. > Getting back to your user space idea, I'd like to understand further to see if I can make some contribution. > Actually I can't think out how to track (there is not timecouner like in kernel) in a easy way, and I have some concerns too. Maybe I was not clear before. You can implement the virtual clocks in the kernel. User space will not need to be involved. It is easy for the kernel to hide the physical clock when virtual clocks are created from it. Thanks, Richard
> -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: 2021年5月15日 1:45 > To: Y.b. Lu <yangbo.lu@nxp.com> > Cc: netdev@vger.kernel.org; David S . Miller <davem@davemloft.net>; Claudiu > Manoil <claudiu.manoil@nxp.com>; Jakub Kicinski <kuba@kernel.org> > Subject: Re: [net-next 0/6] ptp: support virtual clocks for multiple domains > > On Fri, May 14, 2021 at 06:41:28AM +0000, Y.b. Lu wrote: > > I give up supporting physical clock and the timecounters adjusting at the > same time, but I may continue to support virtual clock per your suggestion. > > Okay, so the physical clock stays free running when virtual clocks are active. Yes. > > > Getting back to your user space idea, I'd like to understand further to see if I > can make some contribution. > > Actually I can't think out how to track (there is not timecouner like in kernel) > in a easy way, and I have some concerns too. > > Maybe I was not clear before. You can implement the virtual clocks in the > kernel. User space will not need to be involved. > Thank you very much to make it clear for me. > It is easy for the kernel to hide the physical clock when virtual clocks are > created from it. > > Thanks, > Richard