external-dns/docs/tutorials/webhook-provider.md
Ivan Ka 19263ee9f0
fix(webhook): connection pool leaks, stuck goroutines on server hang, and retry correctness (#6279)
* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

* fix(webhook): connection leaks, goroutine leaks, and retry correctness

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>

---------

Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>
2026-03-20 16:04:14 +05:30

7.2 KiB

Webhook provider

The "Webhook" provider allows integrating ExternalDNS with DNS providers through an HTTP interface. The Webhook provider implements the Provider interface. Instead of implementing code specific to a provider, it implements an HTTP client that sends requests to an HTTP API. The idea behind it is that providers can be implemented in separate programs: these programs expose an HTTP API that the Webhook provider interacts with. The ideal setup for providers is to run as a sidecar in the same pod of the ExternalDNS container, listening only on localhost. This is not strictly a requirement, but we do not recommend other setups.

Architectural diagram

Webhook provider

API guarantees

Providers implementing the HTTP API have to keep in sync with changes to the JSON serialization of Go types plan.Changes, endpoint.Endpoint, and endpoint.DomainFilter. Given the maturity of the project, we do not expect to make significant changes to those types, but can't exclude the possibility that changes will need to happen. We commit to publishing changes to those in the release notes, to ensure that providers implementing the API can keep providers up to date quickly.

Implementation requirements

The following table represents the methods to implement mapped to their HTTP method and route.

Provider endpoints

Provider method HTTP Method Route Description
Negotiate GET / Negotiate DomainFilter
Records GET /records Get records
AdjustEndpoints POST /adjustendpoints Provider specific adjustments of records
ApplyChanges POST /records Apply record

OpenAPI spec is here.

ExternalDNS will also make requests to the / endpoint for negotiation and for deserialization of the DomainFilter.

The server needs to respond to those requests by reading the Accept header and responding with a corresponding Content-Type header specifying the supported media type format and version.

The default recommended port for the provider endpoints is 8888, and should listen only on localhost (ie: only accessible for external-dns).

NOTE: only 5xx responses will be retried and only 20x will be considered as successful. All status codes different from those will be considered a failure on ExternalDNS's side.

NOTE: the --webhook-provider-read-timeout and --webhook-provider-write-timeout flags control the outbound HTTP client timeout. The total client timeout is the sum of both values and covers the full round-trip: writing the request body, waiting for the response, and reading the response body. Requests that exceed this deadline are cancelled and treated as a failure.

Exposed endpoints

Provider method HTTP Method Route Description
K8s probe GET /healthz Used by livenessProbe and readinessProbe
Open Metrics GET /metrics Optional endpoint to expose Open Metrics

The default recommended port for the exposed endpoints is 8080, and it should be bound to all interfaces (0.0.0.0)

Custom Annotations

The Webhook provider supports custom annotations for DNS records. This feature allows users to define additional configuration options for DNS records managed by the Webhook provider. Custom annotations are defined using the annotation format external-dns.alpha.kubernetes.io/webhook-<custom-annotation>.

Custom annotations can be used to influence DNS record creation and updates. Providers implementing the Webhook API should document the custom annotations they support and how they affect DNS record management.

Best practices for webhook provider authors

Status codes

Use the correct HTTP status codes — they directly control ExternalDNS retry behaviour:

Situation Status code Effect
Success (Records, AdjustEndpoints) 200 OK Accepted
Success (ApplyChanges) 204 No Content Accepted
Transient error (rate limit, upstream timeout, etc.) 5xx Retried by ExternalDNS
Permanent error (bad request, auth failure, etc.) 4xx Not retried; logged as failure
Redirects 3xx Treated as a permanent failure; do not use

Response bodies and connection reuse

ExternalDNS drains response bodies before closing them so that TCP connections can be returned to the pool and reused. To keep this effective:

  • Always write a complete response body, even for errors. An empty JSON object {} or a plain-text message is fine.
  • Keep error response bodies small (well under 1 MiB). ExternalDNS caps the drain at 1 MiB; bodies larger than that cause the connection to be discarded rather than pooled, increasing latency and resource usage on both sides.
  • Do not stream indefinitely. Finish writing the response and close it promptly.

Timeouts and cancellation

ExternalDNS propagates request context to all outbound calls. When the controller shuts down or a request times out, the in-flight HTTP connection is cancelled. Providers should:

  • Handle abrupt connection drops gracefully — do not treat a cancelled request as a reason to roll back partially applied changes without verifying state first.
  • Respond within the deadline configured by --webhook-provider-read-timeout and --webhook-provider-write-timeout (default values apply when unset). Long-running DNS API calls should have their own internal timeout shorter than the ExternalDNS deadline.

Memory and goroutine hygiene

  • Close request bodies after reading them to avoid goroutine leaks on the webhook provider side.
  • Avoid holding references to decoded request payloads longer than needed; plan.Changes and endpoint slices can be large for zones with many records.

Provider registry

To simplify the discovery of providers, we will accept pull requests that will add links to providers in this documentation. This list will only serve the purpose of simplifying finding providers and will not constitute an official endorsement of any of the externally implemented providers unless otherwise stated.

Run an ExternalDNS in-tree provider as a webhook

To test the Webhook provider and provide a reference implementation, we added the functionality to run ExternalDNS as a webhook. To run the AWS provider as a webhook, you need the following flags:

- --webhook-server
- --provider=aws
- --source=ingress

The value of the --source flag is ignored in this mode.

This will start the AWS provider as an HTTP server exposed only on localhost. In a separate process/container, run ExternalDNS with --provider=webhook. This is the same setup that we recommend for other providers and a good way to test the Webhook provider.