« Blog Index
February 20, 2018

Caddy 0.10.11 Released with Distributed Auto-HTTPS and Service Discovery

By Matt Holt

Nearly five months after the last release, we're happy to announce that Caddy 0.10.11 is now available! This release brings service discovery for reverse proxying, reusable snippets to the Caddyfile, support for automatic HTTPS in a cluster, improved TLS management at scale, and much more. In fact, almost all the changes in this release are worth highlighting here, so get ready!

Release Cadence and Caddy's Reliability

This is the longest we've gone without a new release (the previous being about 4 months). This was partly as an experiment, to see how robust Caddy is without frequent updates, and to observe how much demand there is from the community for new features and fixes. I also took a break from open source development for a few months.

I'm pleased to report that the results are positive. The community is still quite passionate about Caddy's development, especially with regards to security, the experimental QUIC implementation, and Caddy's documentation.

During the months without a new release, Caddy successfully, autonomously survived several upstream outages that disrupted all other major Let's Encrypt clients. In everything from regional OCSP responder outages (and global ones) to disabling the TLS-SNI challenge completely, only Caddy pulled through all of them without incurring downtime for your sites or requiring you to intervene.

In fact, when the TLS-SNI challenge was disabled, major ACME clients including Traefik, Go's autocert library, and even EFF's flagship client Certbot were impacted to the point where users had to deploy software updates before new certificates were needed, or risk going red. The vast majority of Caddy sites happily sailed through without any adverse effects. This is because Caddy used either HTTP-01 or TLS-SNI-01 by default, not just one. (We hope a replacement TLS-SNI challenge will be found.)

We maintain that Caddy's automatic HTTPS implementation is the most robust, most mature, lowest-maintenance feature set of its kind in the world, and we strongly advocate its use where uptime matters to you in a world of growing reliance on HTTPS. We credit this largely to Caddy being the first to do this, and a passionate community that has tested and vetted Caddy's automatic HTTPS in a variety of scenarios over the longest period of time of all other comparable clients.

While no software is perfect, Caddy clearly stands out among its peers as a resilient technology as the modern Web moves forward.

Automatic HTTPS in Fleet Configurations

As you know, Caddy manages certificates for you by default. It obtains the ones it needs, renews them before they expire, and does so automatically in the background. At scale, this is a bit tricky: when you have multiple Caddy instances managing certificates for the same hostnames, you can run into CA rate limits because each instance may perform a renewal for the same certificate. That is, until now.

With this release, Caddy now synchronizes the renewal of certificates with other instances as long as the instances share the same folder on disk. In other words, Caddy will automatically coordinate the renewal and reloading of certificates with all other instances that share the same .caddy/acme folder. The typical way of achieving this across machines would be to mount the shared folder as part of the local file system and set $CADDYPATH to point to the folder containing the .caddy/acme folder.

Diagram of flow of Caddy's fleet renewal synchronization Diagram of fleet renewal process. A Caddy instance notices that a certificate it has loaded into memory is expiring soon. It checks storage and notices that that copy is expiring too, so it obtains a lock and performs the renewal with Let's Encrypt. It saves the new certificate to disk, and then reloads it into memory for itself. As other instances notice that their loaded copy of the old certificate is expiring, they also check on disk. But they see that it has already been renewed, so they simply load it. Thus only one instance ever performs the renewal, but all can share in it.

For this to work behind a load balancer, you must use the DNS challenge to guarantee that your load balancer won't interfere by routing ACME validation connections to the wrong Caddy instance. (This is not any different from what we have always recommended when using a load balancer.) In the future, we may be able to eliminate the requirement.

Improved TLS Management

In addition, we've improved Caddy's internal bookkeeping with regards to TLS configurations and certificates. If two sites are defined on the same listener with the same hostname, but have conflicting TLS configurations, we now raise an error, since the two configurations cannot be disambiguated at handshake-time from SNI alone. One would always have to be chosen over the other, which is confusing and unexpected.

Some Caddy instances manage thousands of certificates and thousands of sites, with frequent configuration reloads to add or remove sites. Caddy's in-memory certificate cache did not handle this situation well. With this release, certificates are not duplicated in memory, and the cache is flushed when reloads happen, so certificates no longer being used will be deallocated.

It also used to be that if certificates overlapped in the names they served, it was non-deterministic which certificate was used to service a name. Now, certificates are confined to the site they are configured for, rather than a single, global name->certificate mapping.

As a side-effect of how this was implemented, clients can request a specific certificate (chain) if they know its SHA-256 checksum by putting it, hex-encoded, into the SNI field of the ClientHello. (We don't recommend clients to rely on this behavior since it may or may not be RFC-compliant. In fact, probably best to not do it at all. But it's kind of interesting.)

When SNI is used by a client, Caddy will serve a TLS alert, rather than a certificate for the wrong hostname. This behavior is more correct for clients that support SNI. When a client does not support SNI, a default certificate is still served, in an attempt to maintain some backwards compatibility. We may remove this fallback as SNI usage continues to improve.

Service Discovery for Reverse Proxy and FastCGI

It's a great day for those using Caddy with microservices! Caddy now supports SRV upstreams on both the proxy and fastcgi directives, so you can load balance between multiple backends dynamically with DNS. By specifying srv:// or srv+https:// as the scheme for a backend, you can use Caddy natively with your orchestration tool to dynamically proxy to various microservice backends. Thanks to contributions from Mohammad Gufran who made this possible!

Reusable Caddyfile Snippets

Have you ever split your Caddyfile into separate files so that you could use import to avoid repeating parts of it over and over? Happy day for you -- this hack is no longer required. We have added snippets to the Caddyfile syntax. You can define a snippet by defining a site block with parentheses around the single-token key name. They work like this:

(mysnippet) { limits { header 100KB body 1MB } errors errors.log { rotate_size 10 rotate_age 90 } }
sub1.example.com { import mysnippet ... }
sub2.example.com { import mysnippet ... }

In this example, the two sites will have limits on the acceptable header and body sizes as well as error logging with custom rolling options. Using import with snippets makes it much easier to share configuration across many different sites! We are grateful to Craig Peterson for contributing this gem of a feature.

Updated QUIC

The latest version of QUIC is at your fingertips. Google updates its QUIC versions somewhat frequently, and Chrome likewise updates its support in succession, dropping old versions of QUIC. Well for now the latest version of Google's QUIC protocol is working again. Caddy remains the ONLY open source, production-ready web server with built-in QUIC support.

As Google's QUIC protocol continues, it looks like it will converge with the IETF's standard, so we look forward to a single QUIC specification in the next couple years.

More Secure On-Demand TLS

Caddy's On-Demand TLS feature is one-of-a-kind. It obtains and renews certificates "on demand" -- meaning, during a TLS handshake that requires it. This has powerful advantages, like when you may be serving a myriad of domain names dynamically. On-Demand TLS can work where wildcard certificates don't. One obvious downside is that it is easy for malicious clients abuse this feature. That's why, to enable on-demand TLS, you have to specify the maximum number of certificates allowed to be issued. And internal rate limits apply, especially for failed issuances.

Today, though, there's an even better way to enable on-demand TLS. You can now specify an ask URL in your tls directive, and Caddy will send a GET request to that URL with ?domain= in the query string containing the value of the SNI name being requested. Your backend can then approve or deny a certificate for that name. This way you don't need to limit how many certificates can be issued -- as long as you run a gatekeeper that can check before Caddy requests a certificate.

It's easy to use:

tls { ask http://localhost:6007/ondemand }

Then On-Demand TLS is enabled, and your backend will receive queries like:

http://localhost:6007/ondemand?domain=example.com

And if your endpoint returns a status of 200, Caddy will request a certificate if it doesn't already have one. Redirects are not followed. If you return anything other than a 2xx status, Caddy will not allow the client to invoke an ACME exchange.

Pretty cool, huh! This feature is exclusive to Caddy and super-handy in advanced and large-scale use cases.

Changed Signal Trap Behavior

We've swapped the behavior of SIGTERM and SIGQUIT to more easily integrate with common Unix/POSIX workflows and existing tools. For example, kill $CADDY_PID will send TERM which is now a more graceful shutdown.

Caddy ignores HUP now, since the previous behavior of that signal trap was relatively useless. And since HUP was not always deliberately sent by the sysadmins, you had to run Caddy with nohup if you wanted to ignore it. Now you won't need nohup; Caddy will ignore HUP.

9 More DNS Provider Plugins

By default, Caddy can obtain certificates for your domains without any explicit configuration. Sometimes, this is difficult or impossible in certain network configurations. The DNS challenge does not require the external CA to contact your server from the outside; all you have to do is supply credentials in environment variables for your DNS provider account and Caddy will get you certificates that way.

This release, we've added new plugins for 9 providers: AuroraDNS, CloudXNS, DNS Made Easy, Gandi v5, GoDaddy, Lightsail, NS1, Open Telekom Cloud, and PowerDNS. Caddy (and its ACME library, lego) now supports 24 providers in total, more than any other ACME client, with more on the way!

Creating a DNS provider plugin takes just about 1 minute, especially if lego already supports it. Let your DNS provider know how awesome this integration is!

Placeholder for Response Headers

For years now, you've been able to use {>Field-Name} placeholders in your Caddyfile for request header fields. Thanks to a last-minute contribution by Toby Allen, now you can easily use {<Field-Name} for values of response header fields.

Bug Fixes

We've fixed quite a few bugs, with the help of the community and several collaborators. Thanks for your patience and for contributing to the project. We have passed over 200 contributors to the project! A lot of work goes on behind the scenes to test, discuss, and think about the best solutions, so thank you for that.

Many of these improvements would not have happened if companies were not using Caddy seriously within their organization and pushing it to its limits. We thank them for doing that. If your business uses Caddy (or should be using it), we strongly recommend following suit and purchasing commercial licenses and extended support to ensure the best for your company's web services.

I want to extend a special thank-you to Matthew Fay for his outstanding help tackling so many questions so thoroughly in our forum. If he has helped you, consider paying it forward to others in our community!

What's Coming

Currently in the pull request pipeline:

Well, those are the main highlights for what is immediately coming. Feel free to get involved, and help make Caddy, and the Web, better!


« Blog Index