* fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> * fix(webhook): connection leaks, goroutine leaks, and retry correctness Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com> --------- Signed-off-by: ivan katliarchuk <ivan.katliarchuk@gmail.com>
7.2 KiB
Webhook provider
The "Webhook" provider allows integrating ExternalDNS with DNS providers through an HTTP interface.
The Webhook provider implements the Provider interface. Instead of implementing code specific to a provider, it implements an HTTP client that sends requests to an HTTP API.
The idea behind it is that providers can be implemented in separate programs: these programs expose an HTTP API that the Webhook provider interacts with.
The ideal setup for providers is to run as a sidecar in the same pod of the ExternalDNS container, listening only on localhost. This is not strictly a requirement, but we do not recommend other setups.
Architectural diagram
API guarantees
Providers implementing the HTTP API have to keep in sync with changes to the JSON serialization of Go types plan.Changes, endpoint.Endpoint, and endpoint.DomainFilter.
Given the maturity of the project, we do not expect to make significant changes to those types, but can't exclude the possibility that changes will need to happen.
We commit to publishing changes to those in the release notes, to ensure that providers implementing the API can keep providers up to date quickly.
Implementation requirements
The following table represents the methods to implement mapped to their HTTP method and route.
Provider endpoints
| Provider method | HTTP Method | Route | Description |
|---|---|---|---|
| Negotiate | GET | / | Negotiate DomainFilter |
| Records | GET | /records | Get records |
| AdjustEndpoints | POST | /adjustendpoints | Provider specific adjustments of records |
| ApplyChanges | POST | /records | Apply record |
OpenAPI spec is here.
ExternalDNS will also make requests to the / endpoint for negotiation and for deserialization of the DomainFilter.
The server needs to respond to those requests by reading the Accept header and responding with a corresponding Content-Type header specifying the supported media type format and version.
The default recommended port for the provider endpoints is 8888, and should listen only on localhost (ie: only accessible for external-dns).
NOTE: only 5xx responses will be retried and only 20x will be considered as successful. All status codes different from those will be considered a failure on ExternalDNS's side.
NOTE: the --webhook-provider-read-timeout and --webhook-provider-write-timeout flags control the outbound HTTP client timeout.
The total client timeout is the sum of both values and covers the full round-trip: writing the request body, waiting for the response,
and reading the response body. Requests that exceed this deadline are cancelled and treated as a failure.
Exposed endpoints
| Provider method | HTTP Method | Route | Description |
|---|---|---|---|
| K8s probe | GET | /healthz | Used by livenessProbe and readinessProbe |
| Open Metrics | GET | /metrics | Optional endpoint to expose Open Metrics |
The default recommended port for the exposed endpoints is 8080, and it should be bound to all interfaces (0.0.0.0)
Custom Annotations
The Webhook provider supports custom annotations for DNS records. This feature allows users to define additional configuration options for DNS records managed by the Webhook provider. Custom annotations are defined using the annotation format external-dns.alpha.kubernetes.io/webhook-<custom-annotation>.
Custom annotations can be used to influence DNS record creation and updates. Providers implementing the Webhook API should document the custom annotations they support and how they affect DNS record management.
Best practices for webhook provider authors
Status codes
Use the correct HTTP status codes — they directly control ExternalDNS retry behaviour:
| Situation | Status code | Effect |
|---|---|---|
Success (Records, AdjustEndpoints) |
200 OK |
Accepted |
Success (ApplyChanges) |
204 No Content |
Accepted |
| Transient error (rate limit, upstream timeout, etc.) | 5xx |
Retried by ExternalDNS |
| Permanent error (bad request, auth failure, etc.) | 4xx |
Not retried; logged as failure |
| Redirects | 3xx |
Treated as a permanent failure; do not use |
Response bodies and connection reuse
ExternalDNS drains response bodies before closing them so that TCP connections can be returned to the pool and reused. To keep this effective:
- Always write a complete response body, even for errors. An empty JSON object
{}or a plain-text message is fine. - Keep error response bodies small (well under 1 MiB). ExternalDNS caps the drain at 1 MiB; bodies larger than that cause the connection to be discarded rather than pooled, increasing latency and resource usage on both sides.
- Do not stream indefinitely. Finish writing the response and close it promptly.
Timeouts and cancellation
ExternalDNS propagates request context to all outbound calls. When the controller shuts down or a request times out, the in-flight HTTP connection is cancelled. Providers should:
- Handle abrupt connection drops gracefully — do not treat a cancelled request as a reason to roll back partially applied changes without verifying state first.
- Respond within the deadline configured by
--webhook-provider-read-timeoutand--webhook-provider-write-timeout(default values apply when unset). Long-running DNS API calls should have their own internal timeout shorter than the ExternalDNS deadline.
Memory and goroutine hygiene
- Close request bodies after reading them to avoid goroutine leaks on the webhook provider side.
- Avoid holding references to decoded request payloads longer than needed;
plan.Changesand endpoint slices can be large for zones with many records.
Provider registry
To simplify the discovery of providers, we will accept pull requests that will add links to providers in this documentation. This list will only serve the purpose of simplifying finding providers and will not constitute an official endorsement of any of the externally implemented providers unless otherwise stated.
Run an ExternalDNS in-tree provider as a webhook
To test the Webhook provider and provide a reference implementation, we added the functionality to run ExternalDNS as a webhook. To run the AWS provider as a webhook, you need the following flags:
- --webhook-server
- --provider=aws
- --source=ingress
The value of the --source flag is ignored in this mode.
This will start the AWS provider as an HTTP server exposed only on localhost.
In a separate process/container, run ExternalDNS with --provider=webhook.
This is the same setup that we recommend for other providers and a good way to test the Webhook provider.
