So I have found solution and everything is running as intended. Even tho was complete nightmare of trial and error until figuring out for someone new to linux. This is how I did it if someone new maybe struggles for something similar:
1. Have to stop pcscd service on host bazzite cause it interferes and blocks distrobox to interact with smart card:
sudo systemctl stop pcscd.socket
sudo systemctl disable pcscd.socket
sudo systemctl stop pcscd
sudo systemctl disable pcscd
2. Set up new udev rule for smart card in /etc/udev/rules.d/
SUBSYSTEM=="usb", ATTR{idVendor}=="xxxx", ATTRS{idProduct}=="xxxx", MODE="0666", GROUP="yourusername"
Replace xxxx with device IDs you can find using lsusb and yourusername with your actuall username
3. In my experience resetting udev rules like this made whole process more reliable:
sudo udevadm control --reload
sudo udevadm trigger
4. Creat ubuntu distrobox and give access to usb and udev rules. Found that ubuntu 22.04 works better then 24.04 as 24.04 changes GID name for some reason which cause some weird mismatch between host bazzite and ubuntu distrobox:
distrobox create --name ubuntu \
--image ubuntu:22.04 \
--volume /dev/bus/usb:/dev/bus/usb \
--volume /etc/udev/rules.d/:/etc/udev/rules.d
5. In distrobox:
sudo apt update
sudo apt install pcscd pcsc-tools
6. Launch pcscd service inside distrobox using simply "pcscd" or try "sudo pcscd" if service doesnt work properly. Test with pcsc_scan if everything works.
And wolla, simple as that, but absolutely terrible experience. Reasons why it didnt work looking back seems obvious now, but incredibly hard where to look when encountering these things first time.