If you have to debug at that level, and you're not designing hardware, things are really bad.
Some years back, Wes Irish at Xerox PARC tracked down one of the great mysteries of early coax Ethernet - why was throughput so much lower than theory predicted? For this, he got both ends of a building-length coax with many machines on it connected to one office, so he could plug both ends into a storage scope. If the waveforms disagreed, somebody was transmitting when they shouldn't. Storage scopes with large storage were rare then. It was an expensive LeCroy unit.
After the end of each Ethernet packet on coax, there is a brief "quiet time", and then the next packet can be sent, beginning with a sync pattern. The hardware detects if what it is sending does not match what it is receiving, which indicates a collision. Both senders stop, wait a random time so they don't collide again, and retry. This is how "carrier sense multiple access - collision detection", or CSMA-CD, works at the hardware level.
This setup revealed that something on the cable was transmitting a single spike after the end of each packet, during the "quiet time". That reset the "quiet time" timer in the network interface, which inhibited the transition to "look for sync" mode. So the next packet would be ignored.
The quiet time timer was at a very low level - software did not see this event.
What came out of looking at the waveforms was the surprising result that the spike during the quiet time was not coming from either the data source or the destination, but from something elsewhere on the cable. The spike was not synchronized to the packet just sent.
With the waveforms for both ends of the cable visible, speed of light lag revealed both that this was happening and where it was coming from, as distance along the cable.
It turned out that several brands of network interface used a part which contained the quiet time timer, the sync recognizer, and the transmitter power controller. When the timer ran out, the device did a state machine transition, and during that transition, for a nanosecond or so, the transmitter turned on. It wasn't supposed to do that. This generated a spike on the cable, resetting every quiet time timer and causing the next packet to be silently ignored by all stations.
The network interface didn't need to be active to do this. Being powered on was sufficient. One device with that part could halve the data rate on a coax Ethernet. Thousands of network interfaces had to be scrapped to fix this.
As an electronics guy, sometimes it is really reasuring to look what happens on the wire, because it bisects your problem space.
E.g. I had a student once who was frantically trying everything on the raspberry pi to fix some motor control script he made. I suggested he should just write a minimum workable code first and then check if there is a signal on the wire. Turns out the wire was not connected.
Especially with hairy problems looking at the actual signals can be useful. Granted, doing so for UDP/Ethernet frames might be overkill, but it is nice to see it can be done.
Wow -- that's an incredible story! (Was this ever formally written up somewhere? Definitely feels like it deserves it!) I am especially curious about using the lag; it sounds like both ends of the cable were plugged into the same scope? That must have felt exhilarating to finally find!
Also, some things never change: the scope that Matt was using for this is (still) an expensive LeCroy unit...
Yes, both ends of the cable were plugged into the same scope. I got to see the scope with the waveform at PARC, because I was asked to take a look at the result.
There was quiet faxing of waveform pictures to certain IC and board manufacturers. Not much publicity. This was in the 1980s. Ethernet was a niche product.
Huh - I believe Wes gave me a flying lesson this summer. That or there's another ex-Xerox PARC guy training out of Palo Alto who looks like him! Cool story :)
Some years back, Wes Irish at Xerox PARC tracked down one of the great mysteries of early coax Ethernet - why was throughput so much lower than theory predicted? For this, he got both ends of a building-length coax with many machines on it connected to one office, so he could plug both ends into a storage scope. If the waveforms disagreed, somebody was transmitting when they shouldn't. Storage scopes with large storage were rare then. It was an expensive LeCroy unit.
After the end of each Ethernet packet on coax, there is a brief "quiet time", and then the next packet can be sent, beginning with a sync pattern. The hardware detects if what it is sending does not match what it is receiving, which indicates a collision. Both senders stop, wait a random time so they don't collide again, and retry. This is how "carrier sense multiple access - collision detection", or CSMA-CD, works at the hardware level.
This setup revealed that something on the cable was transmitting a single spike after the end of each packet, during the "quiet time". That reset the "quiet time" timer in the network interface, which inhibited the transition to "look for sync" mode. So the next packet would be ignored. The quiet time timer was at a very low level - software did not see this event.
What came out of looking at the waveforms was the surprising result that the spike during the quiet time was not coming from either the data source or the destination, but from something elsewhere on the cable. The spike was not synchronized to the packet just sent. With the waveforms for both ends of the cable visible, speed of light lag revealed both that this was happening and where it was coming from, as distance along the cable.
It turned out that several brands of network interface used a part which contained the quiet time timer, the sync recognizer, and the transmitter power controller. When the timer ran out, the device did a state machine transition, and during that transition, for a nanosecond or so, the transmitter turned on. It wasn't supposed to do that. This generated a spike on the cable, resetting every quiet time timer and causing the next packet to be silently ignored by all stations.
The network interface didn't need to be active to do this. Being powered on was sufficient. One device with that part could halve the data rate on a coax Ethernet. Thousands of network interfaces had to be scrapped to fix this.