@walmis Hi, I'm still trying to debug the DCS freeze issue even though that I have improved all of t

@walmis Hi, I'm still trying to debug the DCS freeze issue even though that I have improved all of the ground and power supply connections. The USB controller is now directly connected to a 24V lab power supply and the freeze issue still persists. I use a program called Wireshark to capture the USB traffic in order to understand the underline activities that cause DCS to freeze. I notice on the captured data that at some point the URB interrupt from the host (i.e. DCS) is not acknowledged by the USB controller. Maybe either something times out or prolonged waiting on the ACK causes DCS to hang. Funny enough, unplugging the USB cable to the USB controller after freezing forces the corresponding ACK's to be finally sent so these ACK's have been in the buffer all along and just didn't get sent timely. Have you seen anything like this before? Please refer to the screenshots and captured data for detail. The relevant data packets are:

USB Host to Stick URB_INTERRUPT
64992 46.373221 IRP ID:0xffffd782f98bc010
66728 47.427333 IRP ID:0xffffd782fe1196e0
USB Stick to Host URB_INTERRUPT
78621 61.816345 IRP ID:0xffffd782f98bc010
78622 61.816350 IRP ID:0xffffd782fe1196e0

Any suggestion is welcome!

HHomeOfMomo @walmis Hi, I'm still trying to debug the DCS freeze issue even though that I ha...

walmis•12/6/22, 5:45 AM

Ah Sh!t these USB issues are hard to debug. And yours is the first such case to my knowledge. Did you try running it through a usb hub? Also could you check the FFB board revision number for me?

Wwalmis Ah Sh!t these USB issues are hard to debug. And yours is the first such case to ...

HomeOfMomoOP•12/6/22, 7:21 AM

I'm sorry to ruin your day. I'm like a magnet that seems to attract outliers. Anyway, I did try using a USB hub earlier as you suggested and observed pretty much the same freeze symptom. The revision of my Rhino FFB board is Rev. 1.2, dated 2022.06.02.

walmis•12/6/22, 7:22 AM

Thanks, I'll do some stress testing by flooding that endpoint with data.

walmis•12/6/22, 7:23 AM

Hopefully will be able to reproduce it

HomeOfMomoOP•12/6/22, 7:27 AM

HomeOfMomoOP•12/6/22, 7:28 AM

@walmis Thank you!

walmis•12/6/22, 7:48 AM

Perhaps your PC mainboard has some USB ports on a different controller ? Like separate USB 3.0 and USB 3.1

walmis•12/6/22, 7:48 AM

This one is head scratcher

walmis•12/6/22, 7:50 AM

@kevjuan

walmis•12/6/22, 9:58 AM

@kevjuan Here's a Python script to stress test that HID endpoint, uploads data at max rate. Let me know if you can reproduce the hangup.

hid_out_stress.zip76.81KB

walmis•12/6/22, 5:31 PM

On my windows machine after 26M transfers still chugging along nicely

Wwalmis @kevjuan Here's a Python script to stress test that HID endpoint, uploads data a...

HomeOfMomoOP•12/6/22, 9:08 PM

My motherboard provides USB 2.0, USB 3.2 Gen 1, and USB 3.2 Gen 2. So far, I only tried on multiple USB 3.2 Gen 1 ports and saw the issue. For some reason, I couldn't get your stress test program started. I kept getting this write error.

HomeOfMomoOP•12/6/22, 9:08 PM

HomeOfMomoOP•12/7/22, 3:09 AM

@walmis When I enter 'dev.get_error()', I get 'WriteFile: (0x00000057) The parameter is incorrect.'

Hypoxia•12/7/22, 7:16 AM

? The emergency button must be : 3 Pin 1no 1nc OR 6 Pin 2no 2nc ??? Thanks for your support

walmis•12/7/22, 7:17 AM

The first one, it should just break the positive power supply line when activated

HHomeOfMomo My motherboard provides USB 2.0, USB 3.2 Gen 1, and USB 3.2 Gen 2. So far, I onl...

walmis•12/7/22, 7:49 AM

gotcha, change the report_id=0x55 to report_id=106
should work

Wwalmis gotcha, change the report_id=0x55 to report_id=106 should work

HomeOfMomoOP•12/7/22, 8:08 AM

Yep, report_id=106 does work. I'll let you know later if something happens. Thanks!

Wwalmis The first one, it should just break the positive power supply line when activate...

Hypoxia•12/7/22, 9:04 AM

You mean it's better to order the second one ?

HHypoxia You mean it's better to order the second one ?

walmis•12/7/22, 9:25 AM

3-pin

Hypoxia•12/7/22, 9:26 AM

HomeOfMomoOP•12/8/22, 7:02 PM

@walmis I got to play with your USB stress test program a bit more. I found out it could go for a long long time when the stick (i.e. motors) was not under any load. As soon as I moved the stick around, the USB error started to occur. Interestingly enough, the Wireshark capture shows the ACK from the endpoint was eventually sent but after 1 second instead of the typical 1 ms. The error check shows 'hid_write/WaitForSingleObject: (0x000003E5) Overlapped I/O operation is in progress.'. Apparently, time-out happens after the long wait, hence, the error. What do you think may cause this kind of delay in ACK from the endpoint?

HomeOfMomoOP•12/8/22, 7:03 PM

walmis•12/8/22, 7:25 PM

One quick thing you could try is to isolate the PSU ground from the mains earth connection, other than that I'm baffled. Looked the code, USB driver is pretty simple, maybe errors occur on cable and are not recoverable. I could send you another board with a different MCU to test.

walmis•12/8/22, 7:27 PM

Wwalmis One quick thing you could try is to isolate the PSU ground from the mains earth ...

HomeOfMomoOP•12/8/22, 8:18 PM

Is the power supply (+19 to 24V) ground tied to USB ground through a 0ohm resistor somewhere on the PCB that I can remove? If not, how do I separate USB ground from power supply ground? Thanks!

Wwalmis One quick thing you could try is to isolate the PSU ground from the mains earth ...

HomeOfMomoOP•12/9/22, 4:13 AM

Thank you for your offer! Let me play with the current board some more. I actually enjoy learning stuff about the USB interface that I knew nothing about before. If, say, I still struggle in a few days, I'll take up on your offer. I'm willing to pay for the shipping to offset some of your cost.

HHomeOfMomo Is the power supply (+19 to 24V) ground tied to USB ground through a 0ohm resist...

walmis•12/9/22, 7:39 AM

You can completely disconnect the 20V (both positive and negative) from the USB board (the fan will not work then though) - that will break the GND loop.
Or 2. remove the GND finger on the mains plug/or the earth wire from the mains lead that connects to the power supply input/or use a mains cable without grounding.

Wwalmis You can completely disconnect the 20V (both positive and negative) from the USB ...

HomeOfMomoOP•12/10/22, 1:39 AM

My +19 to 24V supply is a lab power supply, which has red(+), black(-), and green(chassis ground) terminals. My understanding is, since I'm connecting through red and black, I don't think the earth ground is part of the ground connection. Anyway, I did try tying +19 to 24V directly to the motors bypassing the FFB USB control board as you suggested and I still got the same USB issue. Looking closely at my build, I did notice the USB panel-mount connector that I got through Aliexpress does not have the shield tied to ground. I don't think it would matter but I would try to connect it to ground just to be sure.

HHomeOfMomo My +19 to 24V supply is a lab power supply, which has red(+), black(-), and gree...

walmis•12/10/22, 3:51 PM

Alright, it's not GND loop then, perhaps the USB wires go parallel and close with the power wires?
You could try to add a ferrite to the USB wires. Or maybe try another power source? Man, you have a strange case there

Wwalmis Alright, it's not GND loop then, perhaps the USB wires go parallel and close wit...

HomeOfMomoOP•12/12/22, 5:16 AM

Man, I agree with you that this is a real doozy. I tried AC outlets from different part of my house and from different circuit breakers. I also found a USB cable that had a built-in ferrite core and tried. All of the tried cases so far reproduce the same USB issue (occasional 1 sec ACK from endpoint). I haven't tried tying the shield of the panel-mount USB connector to ground (a somewhat difficult mod) but I'm skeptical that anything will be different. I'm close to the end of my wit unless you can come up with something else for me to try. I was in a process of building a home-made reflow oven but got distracted and didn't finish. I'm seriously considering completing this reflow oven project so that I can put the FFB control board in to reflow for a while just to see if my luck changes... Sigh...

HHomeOfMomo My +19 to 24V supply is a lab power supply, which has red(+), black(-), and gree...

SwiftVengeance•12/14/22, 1:04 PM

Maybe the issue is the lab power supply. I know Chinese rebranded ones are very noisy. Maybe you can try a different one like a laptop power brick or one that uses a transformer. Worth a try?

SSwiftVengeance Maybe the issue is the lab power supply. I know Chinese rebranded ones are very ...

HomeOfMomoOP•12/14/22, 4:48 PM

I actually started out with a laptop power brick and noticed the issue. I thought it would be better by switching to a lab power supply but the problem persisted regardless of form of supply.

SwiftVengeance•12/14/22, 5:14 PM

Have you tried a different computer?

SwiftVengeance•12/14/22, 5:16 PM

try it on a laptop if you have one, run the same tests. It could be the usb controller if your on amd motherboard.

SwiftVengeance•12/14/22, 5:17 PM

you can also try turning off all settings related to usb power savings/ standby/ sleep in windows and in bios.

SwiftVengeance•12/14/22, 5:18 PM

SSwiftVengeance Have you tried a different computer?

HomeOfMomoOP•12/15/22, 2:12 AM

Thank you for your response! My motherboard is AMD based Asus Tuf Gaming X570 Pro WiFi. I finally had a chance to do the same test on my Intel based laptop and could not reproduce the issue. I updated BIOS and AMD's chipset driver for Windows 10 to the latest and turned off all USB power/standby/sleep/selective suspend settings as you suggested. I still see the same issue. Unfortunately, my laptop is not powerful enough for gaming so I'm pretty much stuck. Where did you hear about this USB thing with AMD motherboards? Is it something that people complained a lot in the past?

SwiftVengeance•12/15/22, 2:13 AM

its a problem with other devices too

SwiftVengeance•12/15/22, 2:14 AM

hp had to send me a special cable for my hp reverb g2 so it doesn't crash all the time specifically for amd motherboard

SwiftVengeance•12/15/22, 2:16 AM

https://www.reddit.com/r/HPReverb/comments/oyk6xd/revision_2_of_the_reverb_g2_cable_and_all_the/

r/HPReverb - Revision 2 of the Reverb G2 Cable and all the Mysterie...

402 votes and 680 comments so far on Reddit