Deleted API Token still works?!

I deleted a User API Key but it is still working.
Surely this isn't expected?
Was this page helpful?