Let's Encrypt for cPanel - One Year On

November 10 2016

Almost exactly a year ago I proposed the idea of integrating Let’s Encrypt into cPanel as a commercial venture to a friend of mine. I strongly suspected that a market opportunity existed, but not at all confident that we would be able to access it. I made my pitch nonetheless:

>>> 2015-11-19
[21:18:17] <alex> every1 in the cpanel community wants lets encrypt so bad, but no1 is working on a WHM plugin for it
[21:18:18] <alex> beta is dec 3
[21:18:26] <alex> if we launch one commercially
[21:18:29] <alex> we will get so many sales its stupid
[21:18:30] <alex> i guarantee it
[21:18:38] <alex> like $25 per plugin ez

In the months leading up to that conversation, I had been keeping my eye out on the Let’s Encrypt blog and private beta. Looking at the reference implementations (not yet usable) and the general buzz in the community, it was fairly evident that it was actually a novel improvement to the current way of doing things and that it would take off in a major way.

Working with cPanel on a daily basis at my day job, the process of selling and managing of SSL certificates was miserable. Despite charging fairly heavily for DV certificates, I suspect we were making losses on them, taking the support and operational back-and-forth into account. I doubt other web hosts were doing much better - the tooling was too basic (I guess, in-part thanks to existing CAs) to make it scalable.

It felt very wrong to me that I could not find any evidence of anybody working on this problem. There was a feature request opened in November 2014 for cPanel to implement Let’s Encrypt, but seemingly no urgency on it. This is a full year prior to my deciding to commit. shrug.

My now-partner, knowing nothing about cPanel nor Go (which we wrote the product in) agreed to the plan, more or less immediately- and the first commit landed in git the following day.

Time to Launch

It took us 17 days and 114 commits to make our very first release on December 6, which was embarrassingly distributed as a tgz archive.

A fair amount of this time was spent strace-reverse-engineering numerous cPanel/WHM APIs. We required some undocumented API behaviour and cPanel made life hell for you if you didn’t want to write your plugin in Perl. There was a real cost to writing in Go, but in my opinion real benefits too:

  • Easy to distribute, self-contained native binary
  • Much easier to write safe code, with a modern and mature standard library
  • Would avoid the possibility of suicide triggered by the futile act of trying to learn Perl

Luckily, the actual surface area of our application was not huge, so the time researching for and developing Go wrappers for the cPanel LiveAPIs and other interfaces was not too bad.

We missed the go-live of Let’s Encrypt itself by a couple of days, but nobody in the market had made a move yet. It would largely stay this way for months - the first peep we heard was from Gandi on January 12th, when they announced integration for their own PaaS customers. This was no risk to us.

We needed a website to do sales and documentation - so we knocked one together in 4 hours with a mix of Go, Hugo & Stripe. It looks bad and we feel bad, but I do not think it has hurt us in any real way.

Setting a Price

We set our final prices at $30USD for a single-server licence, and $150USD for an unlimited licence (with some restrictions). Later we added $300USD source code licences (which we were nervous about) and also some other minor licences types.

For me personally, the primary factor that influenced the prices at this level was that I was not sure whether we would get any sales. Hell, I was even considering making it open source, on the day of the release.

In the end, I don’t think it was a terrible price point, based on the fact that we have had a healthy sales volume. A lot of customers remarked - “wow, so cheap for what it is!” - which, to be honest, kind of stung. OTOH, we had a few trying to haggle on the price, so we can’t have been that far off.

Setting your price based on fear or anxiety over the quality of your work is definitely no way to live … and after all this time I don’t know of an alternative.

Time to First Sale

We posted in the cPanel feature thread and waited. (At that time, it was not against the rules. I think in response to us, they have forbidden advertising third-party anything. To be honest it was a dick move by them. We are trying to improve their ecosystem while they are dragging their heels, so it feels a little bit like they are competing with us).

Less than an hour later, we got our first sale. I felt a mix of excitement and impending doom.

At that time, we were forcing people to sign up for free 72-hour trials through the website. You’d get a licence file by email. We really expected everybody to get a trial before purchasing.

Not so. This guy purchased our most expensive licence without having so much as downloaded the plugin to his servers. Hell, we did not even know if it even worked outside of limited test and production environments (I had deployed to Serversaurus shortly beforehand for a smoke test).

[20:54:42] <alex> holy fuck he went for  $150
[20:54:44] <alex> i thought we turned it off
[20:54:58] <alex> oh he went for unlim licence
[20:57:17] <eggsampler> ... did he not try it first lol

From there, we had a fairly steady stream of sales.

Advertising in the cPanel feature request got us a LOT of clicks and sales - it was an ideal channel. It ranked high on Google, it described the problem people were having, and in that thread you would find the solution to your problem.

We posted in a couple of other places, including Web Hosting Talk, and tried to answer people’s questions on the internet whenever we noticed the referrals in Google Analytics. I think my partner spent $5.00 on Google ads, but we did not pursue it.

However, overwhelmingly, the sales seemed to drive themselves. I think this was a combination of word-of-mouth and also the fact that our domain name was highly Google-able: letsencrypt-for-cpanel.com

And awaaaaaay we go …

We had essentially zero costs, apart from our time. Serversaurus was sponsoring us by providing web hosting and had sponsored some of my time, and in return the website would link back to them. Our only real cost was the domain name and $20 in MaxMind fraud API credits.

Even including the time we spend supporting our clients, I suspect we come out strongly ahead.

All sales revenue was split 50⁄50 (though this fact was never discussed/brought up).

Although we had our highest income around ~Q1 of 2016, it has provided a very welcome stream of money for us:

Sales volume over the year

Support

I think the biggest surprise to me about providing support is that our customers exhibited extreme gratitude for the product, and patience when something had gone wrong.

It was a pleasant change from the usual client-initiated abuse you sometimes cop when working in operations.

I do not know if this is a perception bias of working on my own project (nobody up the chain to blame, only myself), but this had a positive effect on my stress and life satisfaction generally. A++ would do again.

We had two major fuck-ups that required us to email our customers:

  • Once we implemented the ACME spec wrong and all renewals would fail (this is when Let’s Encrypt changed to the X3 intermediate due to Windows XP reasons)
  • Once we broke our rpmspec and the package would uninstall itself during upgrades

I expected to cop an earful from these, but again to my surprise, we received kind messages of thanks and patience. :S.

I think our support interactions have been highly satisfactory. In my view this is because there is only the two of us - both developers - answering all the emails. Being able to give the correct answer directly is definitely a boon for our customers.

Of course, our time is limited. I have a job and my partner is in university, so we can’t lose too much time on support. Luckily, our product is pretty solid and does not require babysitting. As the developers, were able to immediately identify and improve the parts of the product that caused support burn, and eventually even created a general support one-liner which would catch issues that we didn’t know about - “unknown unknowns”.

The worst part of the support experience is customers who would have generic problems with their cPanel installations and they would manifest themselves in our product’s behaviour. We have spent many hours of our time debugging such problems. Unfortunately, I think it is a fact of life of writing plugins for a behemoth like cPanel.

Ongoing Development

Apart from critical bug fixes (of which there have been few), we generally would look at improving the product over the weekend. These days, we maybe contribute to the code base every second weekend, with releases more infrequent than that.

A major win here was that the MVP of the product had an incredibly tiny surface area, and we did not feel excessive pressure to build out features as a full-time occupation.

As such, we were able to move fairly slowly (at least, compared to the 17 day initial implementation time) and understand our customers’ evolving needs before going down any particular feature-rabbit-hole.

TFW the competition catches up with you

So, it was obvious from day one that cPanel was going to have a competing implementation eventually - it said so in the cPanel feature request thread.

We were fine with it. It was a kind of “well, we had a good run, and we can make a clean break”. I feel like it is also the reason we set prices so low.

However, cPanel (and the rest of the market) seriously dragged their feet.

In December, we thought we would be dead in April.

In April, we thought we would be dead in June.

In July, we thought we would be dead in August. We made a mad rush to feature-match the competing ‘AutoSSL’ implementation that cPanel finally came out with.

August came and went. I know that at least a couple of customers have moved to cPanel’s first party implementation.

Funnily enough though, cPanel did not feature-match us. I think we actually provide a better product in substantial ways compared to the cPanel offering (at least, as it stands today). Maybe we are too small-fry to bother, or their development scheduled is muddled up with other things.

So, it is November, and our sales are still ticking, customers are expanding their deployments, and new customers are choosing us over the default built-in feature.

I’m not sure what the lesson is, though.

Be first? Compete on features?

Lessons Learned

In descreasing order of mattering to me:

  • Having a 50⁄50 business+technical partner is incredibly motivating, satisfying, de-stressing and error/idea/sanity-checking. A second human being that is able to attack any task that you can is indispensable (even if they don’t know anything about the product domain or implementation specifics). This was a particularly important realization for me, because most of my career has been spent as the solo-implementor. I am no longer taking big or long projects if it’s just me by myself. Not worth it/wrong personality for it.
  • Web hosting market is huge and under-exploited. There are so many crap incumbent solutions and opportunities in there, and web hosts are practically falling over themselves to spend money on quality solutions.
  • Don’t let fear and anxiety drive your decision making. (Well .. lesson kind of learned .. I still need to figure out how to apply it). Value your time, have respect for yourself and be proud of your creations.
  • If you see an opportunity to be first, be first. I knew this intellectually, but lol, seeing it in practice is another thing entirely.
  • Go will never let you down <3

I don’t know if this product will survive another year, but the point at which I didn’t care if we stopped getting sales was months and months ago. Right now I am looking forward to spotting the next opportunity.