Caddy and Automatic HTTPS: How SSL Generates Itself

Setting up HTTPS on a web server usually means a series of manual steps: install Certbot, run a command to request the certificate, edit the Nginx or Apache configuration to point at the certificate files, then install a cron job so the certificate is renewed before it expires. Every step has potential for error — forgetting to open port 80 for validation, a wrong path in the configuration, or a cron job silently failing for months until the certificate expires and the site goes down. Caddy takes a different approach: just mention the domain in the configuration, and HTTPS is active immediately with no extra steps. This article discusses what actually happens behind this “magic” — the protocol used, how domains are validated, and how the certificate lifecycle is managed entirely by Caddy itself.

What Is Caddy and Automatic HTTPS

Caddy is an open-source web server and reverse proxy written in Go, known for its “secure by default” philosophy. Unlike Nginx or Apache, which treat HTTP as the normal mode and HTTPS as an additional feature that must be configured manually, Caddy flips this assumption: once you mention a domain name in the configuration, Caddy immediately assumes you want that domain accessed over HTTPS, and it handles everything itself.

This feature is called Automatic HTTPS. It covers four things at once: issuing TLS certificates, installing them on the server, renewing certificates before they expire, and — if configured — automatically redirecting HTTP traffic to HTTPS. All of this happens without a single extra line of configuration for standard use cases.

To understand the scale of the difference, here’s a workflow comparison between the traditional approach and Caddy:

This fundamental difference isn’t just about convenience — it eliminates an entire class of operational bugs that arise from disconnected manual processes.

The Anatomy of the ACME Protocol

Automatic HTTPS in Caddy is built on ACME (Automatic Certificate Management Environment), a standard protocol (RFC 8555) designed for full automation of the TLS certificate lifecycle — from request, domain ownership validation, issuance, to revocation.

ACME was originally developed by the Internet Security Research Group (ISRG) together with the launch of Let’s Encrypt, the free certificate authority (CA) that became the main driver of mass HTTPS adoption on the web. Caddy supports Let’s Encrypt as its default CA, with ZeroSSL as an automatic fallback if Let’s Encrypt can’t be reached. Additionally, Caddy can be configured to use a company’s internal CA (for example via Smallstep CA) as long as that CA implements ACME.

The Role of CertMagic

All of Caddy’s Automatic HTTPS logic actually lives in a separate library called CertMagic, also written by the same team. CertMagic handles:

• Communication with the ACME server (request, status polling, certificate download)
• Secure storage of certificates and private keys on disk
• Scheduling of automatic checks and renewals
• Selection of the most appropriate challenge type for each domain
• Fallback between CAs if one is unavailable

Because CertMagic is an independent Go library, other developers can use it outside Caddy to build Automatic HTTPS into their own Go applications. But in the Caddy context, CertMagic is fully integrated as part of the core, so developers using Caddy don’t need to know these details — just mention the domain, and CertMagic works behind the scenes.

The General ACME Communication Flow

Here’s an overview of the interaction between Caddy (via CertMagic) and the ACME server:

sequenceDiagram
    participant Caddy
    participant ACME as ACME Server (Let's Encrypt)
    participant Validator as Validation Infrastructure

    Caddy->>ACME: Request certificate for example.com
    ACME-->>Caddy: Send challenge (unique token)
    Caddy->>Caddy: Prepare proof per challenge type
    ACME->>Validator: Verify domain ownership proof
    Validator-->>ACME: Verification result
    ACME-->>Caddy: Certificate issued (if valid)
    Caddy->>Caddy: Save cert and install on the server

The key point: the ACME server doesn’t simply trust that Caddy controls the domain. It sends a challenge that must be answered in a specific way, and only then verifies that answer from an external side — this is the domain validation process.

Challenge Types — How Domains Are Validated

The core of the question “how can Caddy generate SSL by itself” is in this section. Domain validation is the mechanism ensuring that only parties who genuinely control a domain (either through server control or DNS control) can get a certificate for it. Without this mechanism, anyone could request a certificate for someone else’s domain.

Caddy supports three types of ACME challenges, and automatically picks one based on the configuration and network conditions.

HTTP-01 — Validation via a File on Port 80

This is the most common method and Caddy’s default for most cases.

How it works:

1. Caddy requests a certificate from the ACME server for the domain example.com.
2. The ACME server replies with a unique token, for example xK3f9a....
3. Caddy computes a “key authorization” — a combination of the token with the thumbprint of Caddy’s own account key — and serves it at a specific path:

http://example.com/.well-known/acme-challenge/xK3f9a...

4. The ACME server, from its own infrastructure (not from Caddy), makes an HTTP request to that URL.
5. If the response content matches what’s expected, the ACME server considers the domain validated — because only the party that truly controls the server behind example.com could serve a file at that path.

flowchart TD
    A[Caddy requests certificate] --> B[ACME sends token]
    B --> C["Caddy serves token at\n/.well-known/acme-challenge/token"]
    C --> D[ACME server fetches the URL externally]
    D --> E{Content matches?}
    E -- Yes --> F[Domain validated]
    E -- No --> G[Validation failed]

HTTP-01 requires port 80 open to the internet during the provisioning process, even though the site will ultimately be accessed via port 443. If port 80 is blocked by a firewall or redirected elsewhere before Caddy can answer the challenge, certificate issuance will fail.

This method can’t be used for wildcard certificates (*.example.com), because the ACME server doesn’t know which subdomain to check — HTTP-01 validation is always specific per hostname.

TLS-ALPN-01 — Validation via a TLS Handshake on Port 443

This method is useful when port 80 isn’t available at all (for example a server that’s only allowed to serve traffic on port 443, or an environment with strict firewall restrictions).

How it works:

1. Caddy requests a certificate as usual, receiving a token from the ACME server.
2. Instead of serving a file over HTTP, Caddy creates a temporary TLS certificate — a special self-signed certificate whose content embeds the token ownership proof, marked with a special extension called acmeIdentifier.
3. Caddy configures its TLS server to serve this temporary certificate, but only if the request comes with a special ALPN protocol named acme-tls/1.
4. The ACME server performs a TLS handshake to example.com:443 while requesting the acme-tls/1 protocol.
5. Caddy responds with the temporary certificate. The ACME server verifies that the certificate’s content matches the token it gave.
6. If it matches, the domain is validated — without a single regular HTTP request happening.

sequenceDiagram
    participant ACME as ACME Server
    participant Caddy

    ACME->>Caddy: TLS handshake to port 443 (ALPN: acme-tls/1)
    Caddy->>Caddy: Create temporary certificate containing token proof
    Caddy-->>ACME: Send temporary certificate
    ACME->>ACME: Verify the acmeIdentifier extension
    ACME-->>Caddy: Validation successful, real cert issued

The advantage of this method is that it doesn’t need port 80 at all — suitable for servers that truly only open port 443. Its drawback is the same as HTTP-01: no wildcard support, and it only works for domains that can be reached directly by the ACME server (the server must have a public IP address resolved from that domain).

DNS-01 — Validation via DNS Records

This is the only method that supports wildcard certificates, and also the only one that doesn’t need any port open to the internet.

How it works:

1. Caddy requests a certificate, this time possibly for a wildcard domain like *.example.com.
2. The ACME server gives a token that must be installed as a TXT record in DNS with a special name:

_acme-challenge.example.com    TXT    "token-value-from-ACME"

3. Caddy — via a configured DNS provider plugin (Cloudflare, Route53, DigitalOcean, and dozens of other providers) — calls that DNS provider’s API to create this TXT record automatically.
4. Caddy waits a bit to make sure DNS propagation finishes, then tells the ACME server the record is ready.
5. The ACME server queries DNS for _acme-challenge.example.com, matching the TXT record value against what’s expected.
6. If it matches, the domain (including all its wildcards) is validated. Caddy then deletes the TXT record since it’s no longer needed.

flowchart TD
    A[Caddy requests wildcard cert] --> B[ACME sends token]
    B --> C[Caddy calls DNS provider API]
    C --> D["Create TXT record\n_acme-challenge.domain"]
    D --> E[Wait for DNS propagation]
    E --> F[ACME queries the TXT record]
    F --> G{Matches?}
    G -- Yes --> H[Wildcard cert issued]
    G -- No --> I[Validation failed / timeout]

DNS-01 is the best choice for servers behind NAT, VPNs, or strict firewalls — because validation happens purely through DNS control, without the ACME server needing to reach the Caddy server directly at all.

Its drawback is setup complexity: Caddy needs API credentials from the DNS provider and the appropriate plugin (Caddy doesn’t include every provider by default — a custom build or an image that includes the plugins is needed).

Comparison of the Three Challenge Types

By default, Caddy tries HTTP-01 first for regular domains. If port 80 can’t be used, Caddy automatically falls back to TLS-ALPN-01. For wildcard domains, Caddy forces DNS-01 because it’s the only valid option under the ACME protocol.

The Complete Flow from Startup to Active Certificate

To understand the full picture, here’s what happens from the moment Caddy starts until a domain is truly accessible over HTTPS:

flowchart TD
    A[Caddy starts, reads configuration] --> B{Domain needs TLS?}
    B -- No --> Z[Serve as regular HTTP]
    B -- Yes --> C{Certificate already in storage?}
    C -- Yes, still valid --> D[Load existing certificate]
    C -- Missing / expired --> E[Start ACME process]
    E --> F[Pick challenge type per conditions]
    F --> G[Run domain validation]
    G --> H{Validation successful?}
    H -- Yes --> I[Download certificate from CA]
    H -- No --> J[Retry with backoff, or fail]
    I --> K[Save certificate to storage]
    K --> D
    D --> L[Install certificate on the TLS server]
    L --> M[Domain accessible via HTTPS]

Some important details from this flow:

Caddy doesn’t wait for all domains to be ready before serving traffic. Each domain is processed independently — if you have five domains in one configuration, Caddy will start serving the domains whose certificates are ready while others are still provisioning.

The default certificate storage is on local disk, usually in $XDG_DATA_HOME/caddy (commonly ~/.local/share/caddy on Linux), organized per CA and per domain. Its contents include the private key, certificate, and metadata like issue and expiry dates. For multi-instance deployments (for example behind a load balancer with several Caddy nodes), this storage can be pointed at a distributed backend like Redis or a database via a storage plugin, so all instances share the same certificates without each one making separate requests to the ACME server.

CA rate limits are an important consideration. Let’s Encrypt limits the number of certificates that can be issued per domain per week (currently 50 certificates per registered domain per week, and failed retries are also limited). This is why sharing storage between instances matters — without it, each instance will try to request a separate certificate for the same domain and can quickly hit the rate limit.

Auto-Renewal — The Behind-the-Scenes Mechanism

Let’s Encrypt certificates have a short validity period: 90 days. This isn’t a weakness, but a deliberate design — short validity reduces the impact if a private key leaks, and forces the renewal process to become an automatic habit rather than an easily forgotten yearly task.

Caddy handles this through a background process that runs while Caddy is active:

1. Caddy periodically checks all the certificates it manages.
2. For each certificate, Caddy calculates the remaining validity period.
3. If a certificate has less than one third of the total validity remaining (for Let’s Encrypt, this means starting renewal around 30 days before expiry), Caddy begins the renewal process — repeating the entire ACME flow from scratch (request, challenge, validation, issuance) for that domain.
4. The new certificate is installed without restarting the server and without downtime — ongoing TLS connections keep using the old certificate until they finish, while new connections immediately use the new certificate once available.

stateDiagram-v2
    [*] --> Active: Certificate issued
    Active --> NearExpiry: Remaining validity < 1/3
    NearExpiry --> Renewing: Auto renewal triggered
    Renewing --> Active: Renewal successful, new cert installed
    Renewing --> RetryBackoff: Renewal failed
    RetryBackoff --> Renewing: Retry with increasing delay
    RetryBackoff --> Expired: All retries failed, cert expired

If all retry attempts fail — for example because the domain no longer points to the correct server, or DNS provider API credentials expired — the certificate will truly expire and the site will show a TLS error to visitors. Caddy logs this error, so monitoring Caddy logs remains important even though the renewal process is automatic.

Caddy’s retry strategy uses exponential backoff — the delay between failed renewal attempts grows longer, so it doesn’t flood the ACME server with repeated requests in a short time, while also giving temporary problems (like network disruptions) time to recover on their own.

Practical Configuration — Caddyfile

This section shows how all the concepts above translate into real configuration via the Caddyfile, Caddy’s native configuration format that’s more concise than the JSON alternative.

Minimal Configuration

Automatic HTTPS is active by default as soon as you mention a domain:

example.com {
    reverse_proxy localhost:3000
}

No extra lines for TLS. Caddy automatically:

• Requests a certificate for example.com via HTTP-01
• Redirects traffic from port 80 to port 443
• Serves traffic via HTTPS to the backend application on port 3000
• Schedules automatic renewal

Overriding the CA — Staging vs Production

When developing or testing configuration, it’s highly recommended to use Let’s Encrypt’s staging environment, because the production environment has strict rate limits that are easily hit if you restart Caddy repeatedly with wrong configurations:

{
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

example.com {
    reverse_proxy localhost:3000
}

Certificates from staging aren’t trusted by browsers (a warning will appear), but that’s normal — the goal is validating that the entire ACME flow works correctly before switching to the production CA.

Setting Up DNS-01 with a Plugin

For wildcard certificates, a DNS provider plugin is needed. Caddy with the Cloudflare plugin (built via xcaddy or a Docker image that includes the plugin) is configured like this:

{
    email [email protected]
}

*.example.com, example.com {
    tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    }
    reverse_proxy localhost:3000
}

The API token is stored as an environment variable, not hardcoded in the configuration file — an important security practice, especially if the Caddyfile is stored in a Git repository.

export CLOUDFLARE_API_TOKEN="secret-token-from-cloudflare"
caddy run

On-Demand TLS — For Dynamic Domains

Another interesting use case is multi-tenant applications where customers can attach their own custom domains (for example platforms like Shopify or Webflow). Explicitly listing every domain in the configuration isn’t practical when there could be thousands of them changing constantly.

On-demand TLS lets Caddy issue a certificate on the spot, exactly when the first request for a new domain arrives:

{
    on_demand_tls {
        ask http://localhost:8080/check-domain
    }
}

:443 {
    tls {
        on_demand
    }
    reverse_proxy localhost:3000
}

The ask endpoint here is important as a safeguard: before Caddy is willing to issue a certificate for an unknown domain, it asks that endpoint whether the domain is genuinely valid and registered in your system. Without this safeguard, anyone could point any domain at your Caddy server and trigger unlimited certificate issuance — potentially hitting CA rate limits or even being abused as an attack vector.

On-demand TLS must be paired with an ask endpoint in production environments. Without this validation, your server becomes vulnerable to abuse that drains your ACME rate limits for unauthorized domains.

Local Development — Caddy as a Local CA

One feature often overlooked is Caddy’s ability to serve HTTPS even for localhost or internal domains like myapp.local, without the manual setup usually done with the mkcert tool.

For local domains, Caddy doesn’t use ACME at all (since ACME only applies to publicly verifiable domains). Instead, Caddy runs its own local Certificate Authority:

1. On first run, Caddy creates a local root certificate and stores it in local storage.
2. Caddy offers to add this root certificate to the operating system’s trust store (needs one-time admin/sudo permission).
3. Once the root certificate is trusted by the system, Caddy can issue “real” certificates (from the browser’s perspective) for any domain you run locally.

localhost {
    reverse_proxy localhost:3000
}

This configuration directly produces valid HTTPS in the browser for https://localhost, without an untrusted certificate warning — very useful for testing features that need a secure context (like Service Workers or WebAuthn) in the development environment.

When Automatic HTTPS Isn’t Suitable or Needs Adjustment

Although Automatic HTTPS is very powerful, there are several scenarios where the default approach needs adjusting or even disabling:

Internal servers without public access. If the Caddy server is only accessible from an internal network (for example a corporate VPN), public ACME like Let’s Encrypt can’t validate the domain via HTTP-01 or TLS-ALPN-01 because the CA can’t reach the server from outside. The solution is DNS-01 (if the domain is public but the server is private), or using an internal CA like Smallstep step-ca, which supports ACME but runs on your own infrastructure.

Needs for certificates from a corporate/internal CA. Some organizations have compliance policies requiring certificates issued by an internal company CA rather than a public CA. Caddy supports this as long as that CA implements the ACME protocol — just point acme_ca at the internal CA’s endpoint.

A load balancer or CDN in front of Caddy already handling TLS termination. If the architecture already uses a CDN (Cloudflare, CloudFront) or a cloud load balancer that already handles TLS certificates at the front layer, Caddy behind it often doesn’t need to handle TLS at all — traffic from the load balancer to Caddy can be plain HTTP within a private network. In this case, Automatic HTTPS can be explicitly disabled:

example.com {
    tls internal
    reverse_proxy localhost:3000
}

The tls internal directive forces Caddy to use the local CA (not public ACME) — suitable for scenarios where encryption is still needed inside a private network but publicly trusted certificates aren’t required.

Rate limits easily hit during repeated testing. As mentioned before, rapid configuration iteration with repeated restarts can trigger Let’s Encrypt production rate limits. Always use the staging CA when developing new configurations.

Multi-instance without shared storage. If you run many Caddy instances for the same domain without a shared storage backend, each instance will try to request an independent certificate — potentially hitting rate limits and creating inconsistent certificates between instances. The solution is shared storage via a plugin like caddy-storage-redis or a similar backend.

Summary

• Automatic HTTPS in Caddy is built on the ACME protocol, run via an internal library called CertMagic, with Let’s Encrypt as the default CA and ZeroSSL as a fallback.
• Domain validation happens through three types of challenges: HTTP-01 (a file on port 80), TLS-ALPN-01 (a special TLS handshake on port 443), and DNS-01 (a TXT record in DNS, the only one supporting wildcards).
• DNS-01 is the best choice for servers behind NAT/firewalls and for wildcard certificates, but requires a DNS provider plugin and API credentials.
• Caddy automatically picks the most appropriate challenge type based on configuration and port availability, with HTTP-01 as the default.
• Auto-renewal is triggered when the remaining certificate validity drops below one third of the total (for Let’s Encrypt, around 30 days before expiry), running without downtime and without server restarts.
• Certificates are stored in local storage by default, but for multi-instance deployments a shared storage backend should be used to avoid rate limits and inconsistency.
• On-demand TLS enables dynamic certificate issuance for multi-tenant applications, but must be paired with an ask endpoint to prevent abuse.
• For local development, Caddy runs its own local CA so localhost and internal domains can have valid HTTPS without extra tools like mkcert.
• Automatic HTTPS can be disabled or adjusted via tls internal for scenarios where TLS termination is already handled by another layer, or when the server lacks the public access ACME needs.