Internet Service Providers generally route traffic based on where it is cheapest for them, not by the shortest path or least latency. The magic of Cloudflare is that they have so many points of presence and good peering presence, that you get on their network quickly, perhaps your ISP is even peered with them. Once you're on their network, Cloudflare generally has more peers/routes available to them, and likely just has a better path.
I don't know of any provider that get "special treatment", some might peer directly with Cloudflare, resulting in better routing/latency, but Cloudflare's peering policy is open. Cloudflare does have Argo Smart Routing, a paid service which tries to use their data to route your connections to origin more optimally based on their data.
300-500ms is a fair bit of latency, I believe it is likely in that case it is just related to your ISP not having the right transit/peers, where Cloudflare does
