Router internet slow with IPv6 enabled - works fine from clients

Little bit more context, it doesn’t seem to be DNS related as that’s snappy:

root@turris:~# time dig -6 repo.turris.cz

; <<>> DiG 9.11.10 <<>> -6 repo.turris.cz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59990
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;repo.turris.cz.			IN	A

;; ANSWER SECTION:
repo.turris.cz.		1730	IN	CNAME	proxy.turris.cz.
proxy.turris.cz.	92	IN	A	217.31.192.69

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Fri Oct 11 23:15:43 BST 2019
;; MSG SIZE  rcvd: 79

real	0m 0.01s
user	0m 0.00s
sys	0m 0.01s