im having a problem with a project in python raspberry pi and VLM

Building an ai robot with raspberry pi and a VLM and camera to detect the environment and interact but the problem is the AI model cannot detect precise x,y coordinates can someone tell me how prompt engineer works to make it work or what to do optimal way to obtain x,y cooridnates of the objects detectred in the image ? Thank you