[SOLVED] Stale DNS records (it's always DNS)

I’ve been tearing my hair out trying to figure out why my ACME-DNS challenges weren’t working for just one of my domains (well, *.riff.cc) Turns out
zorlin@durian ~ $ dig -t TXT _acme-challenge.riff.cc

; <<>> DiG 9.16.48 <<>> -t TXT _acme-challenge.riff.cc
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5823
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.riff.cc. IN TXT

;; ANSWER SECTION:
_acme-challenge.riff.cc. 207 IN TXT "ii9WIXF_GV0Er1U3mPWUavuCxJYOXfcTauVdbwKczFg"
_acme-challenge.riff.cc. 207 IN TXT "lc7b0j0Tcx8hd2UqjpuxEywDcFX1_mhwJfz4M6wCNnc"

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Sat Feb 24 10:46:31 UTC 2024
;; MSG SIZE rcvd: 164
zorlin@durian ~ $ dig -t TXT _acme-challenge.riff.cc

; <<>> DiG 9.16.48 <<>> -t TXT _acme-challenge.riff.cc
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5823
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.riff.cc. IN TXT

;; ANSWER SECTION:
_acme-challenge.riff.cc. 207 IN TXT "ii9WIXF_GV0Er1U3mPWUavuCxJYOXfcTauVdbwKczFg"
_acme-challenge.riff.cc. 207 IN TXT "lc7b0j0Tcx8hd2UqjpuxEywDcFX1_mhwJfz4M6wCNnc"

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Sat Feb 24 10:46:31 UTC 2024
;; MSG SIZE rcvd: 164
There are two “ghost” TXT records that are still being served even though they were deleted a long time ago. If I add a new record to Cloudflare under that name:
zorlin@durian ~ $ dig -t TXT _acme-challenge.riff.cc @1.1.1.1

; <<>> DiG 9.16.48 <<>> -t TXT _acme-challenge.riff.cc @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41513
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.riff.cc. IN TXT

;; ANSWER SECTION:
_acme-challenge.riff.cc. 300 IN TXT "TESTING"
_acme-challenge.riff.cc. 300 IN TXT "ii9WIXF_GV0Er1U3mPWUavuCxJYOXfcTauVdbwKczFg"
_acme-challenge.riff.cc. 300 IN TXT "lc7b0j0Tcx8hd2UqjpuxEywDcFX1_mhwJfz4M6wCNnc"

;; Query time: 10 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sat Feb 24 10:50:54 UTC 2024
;; MSG SIZE rcvd: 184
zorlin@durian ~ $ dig -t TXT _acme-challenge.riff.cc @1.1.1.1

; <<>> DiG 9.16.48 <<>> -t TXT _acme-challenge.riff.cc @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41513
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.riff.cc. IN TXT

;; ANSWER SECTION:
_acme-challenge.riff.cc. 300 IN TXT "TESTING"
_acme-challenge.riff.cc. 300 IN TXT "ii9WIXF_GV0Er1U3mPWUavuCxJYOXfcTauVdbwKczFg"
_acme-challenge.riff.cc. 300 IN TXT "lc7b0j0Tcx8hd2UqjpuxEywDcFX1_mhwJfz4M6wCNnc"

;; Query time: 10 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sat Feb 24 10:50:54 UTC 2024
;; MSG SIZE rcvd: 184
It shows up. If I remove that record, it goes back to the “ghost two”. How the heck do I get rid of the bad records?
5 Replies
Erisa
Erisa4mo ago
On SSL/TLS > Edge certificates, there are no pending certificates, they're all active? _acme-challenge is used by Cloudflare to issue edge certificates but typically we would clean up the records after issuance is complete
wings
wings4mo ago
These records had been there for at least 12 hours There's an Active certificate that expires on 2024-05-15 I disabled Universal SSL to "fix" it which works, but as soon as I re-enable it new records appear that do not go away (even after 5-10 minutes) Waited 5 minutes and that Active certificate is completely gone now... going to wait the full 10 and re-enable Universal Oh thank god. Turned everything back on and the ghost records are still gone.
Erisa
Erisa4mo ago
Phew That must have kicked the process to clean them up
wings
wings4mo ago
Thanks for the help! 🙂
Gustavo Hellwig
Gustavo Hellwig4mo ago
hey, I have the same issue as this. Is anyone with an answer?