If you rarely update that config and you have many requests needing it, then you can have D1 as the source of truth where you write your config updates, and then also put them in Workers KV. Your worker can read from Workers KV when a stale value (up to 60s) is OK for that case, and go to D1 if you need the latest/freshest configuration.
Using the local CDN Cache API from your worker sometimes give you faster lookups than Workers KV as well, but it also depends on your scenario. It's not clear from what you said above.
This design can give you the speed when you can use cached values and have D1 as a fallback for your endpoints that need the latest.