eKristensen
eKristensen
PostsComments
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
I'll wrap up for today. (I'll try to do some more rebooting of my device in between the different drivers tomorrow, maybe that helps) I would like to know what version of OpenCL I should target. Thx 🙂
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
libmali-valhall-g610-g13p0-dummy_v1.9-1-2d267b0_arm64.deb <-- I get no errors, but I think it might be secretly be using the CPU anyways (because I see 50-150% CPU consumption when I use the container stats command) libmali-valhall-g610-g13p0-gbm_v1.9-1-2d267b0_arm64.deb <-- Kernel error firmware reloading long python error msg libmali-valhall-g610-g6p0-dummy_v1.9-1-2d267b0_arm64.deb and libmali-valhall-g610-g6p0-gbm_v1.9-1-2d267b0_arm64.deb <-- segfault
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
so
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
passing firmware made no difference
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
I am using libmali-valhall-g610-g13p0-dummy_v1.9-1-2d267b0_arm64.deb (i replaced the version in the filename to keep track of the variants) WITHOUT passing the firmware to the container
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
It does not say it failed over to CPU
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
It works now, no kernel errors, BUT I also see high CPU consumption when I ask the machine learning container to perform ml tasks
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
I am also seeing this: 
warning: det_size is already set in detection model, ignore
Error: An error occurred attempting to execute a workload: CL error: clGetEventProfileInfo. Error code: -58 at function Execute [/devenv/armnn/src/backends/cl/workloads/ClMultiplicationWorkload.cpp:82]
Error: An error occurred attempting to execute a workload: CL error: clGetEventProfileInfo. Error code: -58 at function Execute [/devenv/armnn/src/backends/cl/workloads/ClMultiplicationWorkload.cpp:82]
Error: An error occurred attempting to execute a workload: CL error: clGetEventProfileInfo. Error code: -58 at function Execute [/devenv/armnn/src/backends/cl/workloads/ClConvolution2dWorkload.cpp:163]
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:160: RuntimeWarning: overflow encountered in multiply
  bbox_preds = bbox_preds * stride
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:160: RuntimeWarning: invalid value encountered in multiply
  bbox_preds = bbox_preds * stride
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:162: RuntimeWarning: overflow encountered in multiply
  kps_preds = net_outs[idx+fmc*2] * stride
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:162: RuntimeWarning: invalid value encountered in multiply
  kps_preds = net_outs[idx+fmc*2] * stride
warning: det_size is already set in detection model, ignore
Error: An error occurred attempting to execute a workload: CL error: clGetEventProfileInfo. Error code: -58 at function Execute [/devenv/armnn/src/backends/cl/workloads/ClMultiplicationWorkload.cpp:82]
Error: An error occurred attempting to execute a workload: CL error: clGetEventProfileInfo. Error code: -58 at function Execute [/devenv/armnn/src/backends/cl/workloads/ClMultiplicationWorkload.cpp:82]
Error: An error occurred attempting to execute a workload: CL error: clGetEventProfileInfo. Error code: -58 at function Execute [/devenv/armnn/src/backends/cl/workloads/ClConvolution2dWorkload.cpp:163]
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:160: RuntimeWarning: overflow encountered in multiply
  bbox_preds = bbox_preds * stride
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:160: RuntimeWarning: invalid value encountered in multiply
  bbox_preds = bbox_preds * stride
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:162: RuntimeWarning: overflow encountered in multiply
  kps_preds = net_outs[idx+fmc*2] * stride
/opt/venv/lib/python3.11/site-packages/insightface/model_zoo/retinaface.py:162: RuntimeWarning: invalid value encountered in multiply
  kps_preds = net_outs[idx+fmc*2] * stride
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
I found this, it maybe be relevant? https://discuss.tvm.apache.org/t/most-tasks-failed-with-autoscheduler-on-mali-g610-gpu/16139/7
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
This one is the g6p0 installed before a reboot
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
So this one is the g13p0 variant (without a reboot)
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
Aug 08 00:44:53 ek-arm kernel: mali fb000000.gpu: Ctx 5_0 Group 0 CSG 0 CSI: 0
                               CS_FATAL.EXCEPTION_TYPE: 0x40 (CS_CONFIG_FAULT)
                               CS_FATAL.EXCEPTION_DATA: 0x0
                               CS_FATAL_INFO.EXCEPTION_DATA: 0x0
Aug 08 00:44:53 ek-arm kernel: mali fb000000.gpu: Ctx 5_0 Group 0 CSG 0 CSI: 0
                               CS_FATAL.EXCEPTION_TYPE: 0x40 (CS_CONFIG_FAULT)
                               CS_FATAL.EXCEPTION_DATA: 0x0
                               CS_FATAL_INFO.EXCEPTION_DATA: 0x0
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
I rebooted. I can see more details about the segfault I am getting in the kernel log
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
Does the orange pi need to be restarted to use a different driver ?
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
which versin of opencl does immich expect ?
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
so to demystifying the name libmali <-- Driver name valhall-g610 <-- Hardware identifier. Orange Pi 5 Plus has ARM Mali-G610 MP4 and the codename is Valhall g6p0 / g13p0 <-- OpenCL version dummy / gbm / etc <-- Driver variant/addons If I am getting it right
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
I think I even found the maybe just a difference between g6p0 and g13p0. One is OpenCL 2.1 the other 3.0 source: https://www.roselladb.com/install-opencl-orangepi5-debian-ubuntu.htm
via https://github.com/Joshua-Riek/ubuntu-rockchip/issues/879
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
It is a really long python trace.
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
Okay I got the really long error again
90 replies
IImmich
Created by eKristensen on 7/31/2024 in #help-desk-support
Orange Pi 5 Plus Rockchip RK3588 Hardware ML Acceleration segfault help wanted
ek-arm kernel: mali fb000000.gpu: Power transition timed out unexpectedly
ek-arm kernel: mali fb000000.gpu:         MCU desired = 1
ek-arm kernel: mali fb000000.gpu:         MCU sw state = 2
ek-arm kernel: mali fb000000.gpu: Current state :
ek-arm kernel: mali fb000000.gpu:         Shader=0000000000000000
ek-arm kernel: mali fb000000.gpu:         Tiler =0000000000000000
ek-arm kernel: mali fb000000.gpu:         L2    =0000000000000001
ek-arm kernel: mali fb000000.gpu:         MCU status = 2
ek-arm kernel: mali fb000000.gpu: Cores transitioning :
ek-arm kernel: mali fb000000.gpu:         Shader=0000000000000000
ek-arm kernel: mali fb000000.gpu:         Tiler =0000000000000000
ek-arm kernel: mali fb000000.gpu:         L2    =0000000000000000
ek-arm kernel: mali fb000000.gpu: Sending reset to GPU - all running jobs will be lost
ek-arm kernel: mali fb000000.gpu: Preparing to soft-reset GPU
ek-arm kernel: mali fb000000.gpu: Wait for MCU power on failed on scheduling tick/tock
ek-arm kernel: mali fb000000.gpu: Resetting GPU (allowing up to 500 ms)
ek-arm kernel: mali fb000000.gpu: Register state:
ek-arm kernel: mali fb000000.gpu:   GPU_IRQ_RAWSTAT=0x00000000   GPU_STATUS=0x00000000  MCU_STATUS=0x00000002
ek-arm kernel: mali fb000000.gpu:   JOB_IRQ_RAWSTAT=0x00000000   MMU_IRQ_RAWSTAT=0x00000000   GPU_FAULTSTATUS=0x00000000
ek-arm kernel: mali fb000000.gpu:   GPU_IRQ_MASK=0x00000000   JOB_IRQ_MASK=0x00000000   MMU_IRQ_MASK=0x00000000
ek-arm kernel: mali fb000000.gpu:   PWR_OVERRIDE0=0x00000000   PWR_OVERRIDE1=0x00000000
ek-arm kernel: mali fb000000.gpu:   SHADER_CONFIG=0x00000000   L2_MMU_CONFIG=0x00000000   TILER_CONFIG=0x00000000
ek-arm kernel: mali fb000000.gpu: reloading firmware
ek-arm kernel: mali fb000000.gpu: Reset complete
ek-arm kernel: mali fb000000.gpu: Power transition timed out unexpectedly
ek-arm kernel: mali fb000000.gpu:         MCU desired = 1
ek-arm kernel: mali fb000000.gpu:         MCU sw state = 2
ek-arm kernel: mali fb000000.gpu: Current state :
ek-arm kernel: mali fb000000.gpu:         Shader=0000000000000000
ek-arm kernel: mali fb000000.gpu:         Tiler =0000000000000000
ek-arm kernel: mali fb000000.gpu:         L2    =0000000000000001
ek-arm kernel: mali fb000000.gpu:         MCU status = 2
ek-arm kernel: mali fb000000.gpu: Cores transitioning :
ek-arm kernel: mali fb000000.gpu:         Shader=0000000000000000
ek-arm kernel: mali fb000000.gpu:         Tiler =0000000000000000
ek-arm kernel: mali fb000000.gpu:         L2    =0000000000000000
ek-arm kernel: mali fb000000.gpu: Sending reset to GPU - all running jobs will be lost
ek-arm kernel: mali fb000000.gpu: Preparing to soft-reset GPU
ek-arm kernel: mali fb000000.gpu: Wait for MCU power on failed on scheduling tick/tock
ek-arm kernel: mali fb000000.gpu: Resetting GPU (allowing up to 500 ms)
ek-arm kernel: mali fb000000.gpu: Register state:
ek-arm kernel: mali fb000000.gpu:   GPU_IRQ_RAWSTAT=0x00000000   GPU_STATUS=0x00000000  MCU_STATUS=0x00000002
ek-arm kernel: mali fb000000.gpu:   JOB_IRQ_RAWSTAT=0x00000000   MMU_IRQ_RAWSTAT=0x00000000   GPU_FAULTSTATUS=0x00000000
ek-arm kernel: mali fb000000.gpu:   GPU_IRQ_MASK=0x00000000   JOB_IRQ_MASK=0x00000000   MMU_IRQ_MASK=0x00000000
ek-arm kernel: mali fb000000.gpu:   PWR_OVERRIDE0=0x00000000   PWR_OVERRIDE1=0x00000000
ek-arm kernel: mali fb000000.gpu:   SHADER_CONFIG=0x00000000   L2_MMU_CONFIG=0x00000000   TILER_CONFIG=0x00000000
ek-arm kernel: mali fb000000.gpu: reloading firmware
ek-arm kernel: mali fb000000.gpu: Reset complete
90 replies