external-dns/docs/tutorials
Thibault Jamet 089744c6ff
Add cache at provider level
**Description**

In the current implementation, DNS providers are called to list all
records on every loop. This is expensive in terms of number of requests
to the provider and may result in being rate limited, as reported in 1293
and 3397.

In our case, we have approximately 20,000 records in our AWS Hosted Zone.
The ListResourceRecordSets API call allows a maximum of 300 items per call.
That requires 67 API calls per external-dns deployment during every sync period

With this, we introduce an optional generic caching mechanism at the provider
level, that re-uses the latest known list of records for a given time.

This prevents from expensive Provider calls to list all records for each
object modification that does not change the actual record (annotations,
statuses, ingress routing, ...)

This introduces 2 trade-offs:

1. Any changes or corruption directly on the provider side will be
longer to detect and to resolve, up to the cache time

2. Any conflicting records in the DNS provider (such as a different
external-dns instance) injected during the cache validity will cause
the first iteration of the next reconcile loop to fail, and hence add a
delay until the next retry

**Checklist**

- [X] Unit tests updated
- [X] End user documentation updated

Change-Id: I0bdcfa994ac1b76acedb05d458a97c080284c5aa
2024-08-14 10:20:56 +02:00
..
akamai-edgedns.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
alibabacloud.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
ANS_Group_SafeDNS.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
aws-load-balancer-controller.md docs: replace mentions of ingress.class annotations for the spec.ingressClassName field 2023-05-05 16:11:26 +02:00
aws-sd.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
aws.md Add cache at provider level 2024-08-14 10:20:56 +02:00
azure-private-dns.md azure-private-dns: Fix LoadBalancer example 2024-08-07 08:05:06 +03:00
azure.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
bluecat.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
civo.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
cloudflare.md Update cloudflare.md 2024-06-29 15:39:52 +02:00
contour.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
coredns.md Add documentation about etcd HTTPS for CoreDNS provider 2024-06-09 08:53:27 +02:00
designate.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
digitalocean.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
dnsimple.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
dyn.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
exoscale.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
externalname.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
f5-virtualserver.md Add F5 VirtualServer source 2023-03-14 10:23:56 +01:00
gandi.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
gateway-api.md docs/gateway-api: update supported API versions 2024-07-14 09:07:17 -07:00
gke.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
gloo-proxy.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
godaddy.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
hostport.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
ibmcloud.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
istio.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
kong.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
kops-dns-controller.md Add documentation on kops-compatibility-mode 2021-06-24 08:55:42 +02:00
kube-ingress-aws.md docs: replace mentions of ingress.class annotations for the spec.ingressClassName field 2023-05-05 16:11:26 +02:00
linode.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
mx-record.md docs: upgrade mkdocs and fix warnings 2024-06-07 08:15:44 +02:00
nginx-ingress.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
nodes.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
ns1.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
ns-record.md docs: upgrade mkdocs and fix warnings 2024-06-07 08:15:44 +02:00
openshift.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
oracle.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
ovh.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
pdns.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
pihole.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
plural.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
public-private-route53.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
rcodezero.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
rdns.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
rfc2136.md Merge pull request #4283 from angeloxx/feature/rfc-2136-ptr 2024-06-20 00:44:17 -07:00
scaleway.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
security-context.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
tencentcloud.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
traefik-proxy.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
transip.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
ultradns.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
vinyldns.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
vultr.md update docs to v0.14.2 2024-05-18 18:32:35 +02:00
webhook-provider.md docs: upgrade mkdocs and fix warnings 2024-06-07 08:15:44 +02:00