Lost Communication with MCU
Hi Gents, got an issue that I'm chasing and I could use some guidance on how to track down the offending piece of hardware.
So I built my V-Core 4 300 Hybrid about 6 months ago, spent a lot of time getting it adjust and tuned, and had it working really well. In addition to the stock parts, I have 6mm PC enclosure panels, a RatPack filter, and my custom Pi HAT (5V power, cooling fan, and USB 2.0 hub). The problem I am facing is that I have added a RatRoaster chamber heater, and when the chamber gets up to temperature (50C is the temperature I've been trying for), the machine shuts down with the error "Lost Communication with MCU", and usually says "Lost Communication with MCU Beacon". However, I have gotten slightly different errors, such as "Lost Communication with MCU toolboard t0", and "Lost Communication with MCU BTT Octopus". Once the chamber cools down slightly, restarting the machine works and the error has disappeared.
The sequence of events goes like this:
1. Begin print with the chamber temperature in the print set to 50C.
2. RatRoaster comes on and begins heating the chamber, the RatRoaster heater core temperature is set to be limited to 115C (same as the bed), and chamber temperature is set to 50C. At this time, RatOS brings the bed to a z-height of about 150mm (middle of the chamber).
3. Once the chamber reaches 50C, RatOS beings the bed down to 100C (the temperature for the print, I'm using ABS), brings the heater core temperature setpoint down to 75C, and raises the extruder to 150C.
4. RatOS begins heat soaking the bed for 15 minutes. This brings the z-height up to 5mm, this is supposed to help heat up the Beacon.
5. The machine generally shuts down during the 15-minute soak period with the above errors.
[More info in next comment]
7 Replies
What I've tried so far:
1. Removed the USB connections on my Pi HAT that go to the chamber camera and the touchscreen. This takes my USB hub out of the mix. The shutdown still occurs.
2. Replaced the EBB42 toolboard, as I had gotten the toolboard error instead of the beacon error on more than one occasion. The shutdown still occurs.
3. Cleared some disk space on the Pi, the main file system wasn't anywhere near full, but the tempfs was close, so I freed space there. The shutdown still occurs.
4. I re-soldered the cable connector solder points on the beacon board. The shutdown still occurs.
Because the shutdown happens only when the chamber is getting heated, I think this must be a hardware fault, and it must be something inside the chamber. I have a new Beacon on order, but it won't be here for a few days.
What I'm looking for:
1. What logs on the Pi can I look at that would shed more light on exactly what communication error is occurring, and from what device? Is there any debugging that can be turned on to further facilitate this?
2. Am I overlooking anything in my analysis? Should I be looking for something not in the chamber?
Thanks in advance for any advice you can provide.
Try blowing a heat gun at different connections. See if you can pinpoint where heat causes the disconnect?
OK, awesome, thanks, I will try that and see what happens. At this point I think it might be either the Beacon itself or the Beacon cable/connector since that's the component that I can see the temperature tracking closest to the shutdown behavior. When the bed gets close to the Beacon during the heat soak is when it happens.
Makes sense then
I replaced the Beacon. It still doesn't work. It shuts down exactly the same as before. I don't know what to try now.
Can someone please tell me how to get debug logs out of the Pi or the Octopus board? I have to get more information so that I know what to look for.
The error messages vary. Each one says "Lost communication with MCU", but the word/phrase after that varies. I've gotten "Lost communication with MCU 'mcu'", "Lost communication with MCU 'toolhead_0'", and "Lost communication with MCU 'beacon'". This is what's confusing me, because it's not the same component every time.
Debug.zip will have everything, the individual logs are easy to find in mainsail. Go to the machine tab, and select logs up top instead of config
I'm made a bit of progress. I've determined that these disconnects I'm experiencing are USB-related. The Pi's logs are showing USB disconnects from a device anytime it happens:
[ 2.389743] hub 1-1.3:1.0: USB hub found
[ 2.478972] usb 1-1.4: new full-speed USB device number 4 using xhci_hcd
[ 2.594810] usb 1-1.4: New USB device found, idVendor=1d50, idProduct=614e, bcdDevice= 1.00
[ 2.596790] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2.678984] usb 1-1.3.1: new full-speed USB device number 5 using xhci_hcd
[ 2.786037] usb 1-1.3.1: New USB device found, idVendor=04d9, idProduct=8030, bcdDevice= 1.00
[ 2.787951] usb 1-1.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2.808065] hid-generic 0003:04D9:8030.0001: input,hidraw0: USB HID v1.10 Device [BIQU BTT-HDMI7] on usb-0000:01:00.0-1.3.1/input0
[ 2.886973] usb 1-1.3.2: new high-speed USB device number 6 using xhci_hcd
[ 3.121138] usb 1-1.3.2: New USB device found, idVendor=1bcf, idProduct=28c5, bcdDevice= 9.24
[ 3.122854] usb 1-1.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3.124580] usb 1-1.3.2: Product: 3DO USB CAMERA V2
[ 3.218969] usb 1-1.3.3: new full-speed USB device number 7 using xhci_hcd
[ 3.328243] usb 1-1.3.3: New USB device found, idVendor=1d50, idProduct=614e, bcdDevice= 1.00
[ 3.329931] usb 1-1.3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3.418961] usb 1-1.3.4: new full-speed USB device number 8 using xhci_hcd
[ 3.535678] usb 1-1.3.4: New USB device found, idVendor=04d8, idProduct=e72b, bcdDevice=20.10
[ 3.537536] usb 1-1.3.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5.757323] cdc_acm 1-1.4:1.0: ttyACM0: USB ACM device
[ 5.775035] cdc_acm 1-1.3.3:1.0: ttyACM1: USB ACM device
[ 5.826408] cdc_acm 1-1.3.4:1.0: ttyACM2: USB ACM device
[ 5.826747] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[ 7.271054] usb 1-1.3.2: Found UVC 1.00 device 3DO USB CAMERA V2 (1bcf:28c5)
[ 7.499272] input: 3DO USB CAMERA V2: 3DO USB CAME as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.3/1-1.3.2/1-1.3.2:1.0/input/input1
[ 8.533407] hid-multitouch 0003:04D9:8030.0001: input,hidraw0: USB HID v1.10 Device [BIQU BTT-HDMI7] on usb-0000:01:00.0-1.3.1/input0
[ 683.921036] usb 1-1.3.4: USB disconnect, device number 8
[ 684.149134] usb 1-1.3.4: new full-speed USB device number 9 using xhci_hcd
[ 684.264401] usb 1-1.3.4: New USB device found, idVendor=04d8, idProduct=e72b, bcdDevice=20.10
[ 684.264425] usb 1-1.3.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 684.278711] cdc_acm 1-1.3.4:1.0: ttyACM3: USB ACM device
root@3dp-rrvcore4:/var/log# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 1d50:614e OpenMoko, Inc. stm32f446xx
Bus 001 Device 009: ID 04d8:e72b Microchip Technology, Inc. Beacon RevH
Bus 001 Device 007: ID 1d50:614e OpenMoko, Inc. stm32g0b1xx
Bus 001 Device 006: ID 1bcf:28c5 Sunplus Innovation Technology Inc. 3DO USB CAMERA V2
Bus 001 Device 005: ID 04d9:8030 Holtek Semiconductor, Inc. BTT-HDMI7
Bus 001 Device 003: ID 0451:8142 Texas Instruments, Inc. TUSB8041 4-Port Hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@3dp-rrvcore4:/var/log#
The times are from the dmesg log. The device that disconnected at 683.92 was the Beacon. However, I've also had the EBB42 toolboard disconnect, as well as the Octopus board. It's not consistent. This tells me that the problem may actually be on the Pi.
One thing that I'm not sure about is that the Pi is showing that the EBB42 Toolboard (device 7, stm32g0b1xx), The Octopus board (device 4, stm32f446xx), the BTT-HDMI7 touchscreen (device 5), and the Beacon (device 8 before the disconnect, device 9 after the disconnect) are all "full-speed" USB devices, which is a USB 1.1 connection. Is that correct? I would think that those devices should connect as "high-speed" USB devices (USB 2.0).
I'm going to replace the Pi to be sure there isn't a fault there.
I have finally tracked down the root cause of the issue. I built my own SSR for powering the chamber heater, and I used a design which is instantaneous switching. The instantaneous switching is causing enough conducted and radiated EMI that it eventually interferes with the USB communication and causes one or more devices to disconnect from the Pi.
I moved the chamber heater's mains supply to a separate 120V circuit in my house, fed through a long extension cord, and placed the rest of the electronics on a Tripp-Lite IsoBar filtering power strip. This temporarily solved the issue by keeping the SSR's switching noise away from the printer's 120V circuit.
I will have to redesign the SSR to use a zero-crossing switching circuit instead of instantaneous switching.