Is startup validation bullshit?

By dan - On

In 2013 I wrote an article called ‘Is startup validation bullshit?’. It was the most popular post I had written at the time and it led to me writing my first book The 7 Day Startup. Since my blog at the time (WP Curve) has been taken down, I found it on archive.org and re-produced it here. I’ll be referring to it in coming weeks as I start working on another SAAS startup. Jump on my emails to follow the journey. I added a few updated thoughts on Twitter today:

July 9 2013 – Is Startup Validation Bullshit? (WP Curve Blog)

Shortly before starting my software app Informly I read the Lean Startup and had lots of conversations with some very smart software entrepreneurs about validation. I also read lots of posts from people who had ‘validated’ their ideas and had great results.

The problem is what happens if your results aren’t as rosy as those examples?

I feel I can offer a unique perspective here because I launched a product (the first version of inform.ly) last year that ultimately failed. It had over 4,000 free signups and only 15 paid customers ($9 / month). I eventually shut down that version of the product. Here are some of the things I learnt.

(The idea was a dashboard that brought together all of your stats into the one place in a simple way).

Validation doesn’t work so well when the answer isn’t an obvious YES

The popular stories you hear about startup validation go something like this:

  • I created a website with a brief video.
  • It went viral.
  • I bought a yacht.

Dropbox started from a 3 minute video posted to Hacker News (more here). As they progressed through the beta they had their signup list jump to 75,000 people in 1 day. Here’s a cool slideshare from their founder Drew Houston.

Dropbox Startup Lessons Learned from gueste94e4c

Well that’s great if you can get 75,000 optins in 1 day sounds like a good idea but if you don’t is it a bad idea? Maybe you just don’t have much kudos on Hacker News?

The reality is that most ideas aren’t going to go viral. Let’s face it, the chances of you coming up with the next dropbox are fairly minimal. But even if your idea is great will a video showing it’s features go viral? There are a lot of factors at play there I wouldn’t be betting on it.

Email optin / beta signup totals don’t work

As with dropbox above and numerous other examples, email signups is often considered the key indicator of whether an idea is good. With the first version of Informly I had:

  • 1,000 people sign up for the beta (3 months)
  • 1,200 people entered their email to be notified of launch (around 7% but that was to the entire site and I was doing a lot of content)

These may seem like small numbers but for someone who had taken 5 years to built up an email list of 2,000 people in my last business I really felt like I was onto a winner.

However as I’ve found with this idea and other versions (more later) there is a very big difference between someone entering their email into a box and someone paying you each month for a product.

People saying it’s a good idea doesn’t mean it is

As part of validating Buffer Joel Gascoigne â€œsimply tweeted the link and asked people what they thought of the idea”. His post on validating Buffer indicates that he read a lot into this including:

“After a few people used it to give me their email and I got some useful feedback via email and Twitter, I considered it validated. In the words of Eric Ries, I had my first validated learning about customers”.

Really? A few of your mates said it was a good idea and therefore it’s ‘validated’? Obviously it turns out that Buffer was a good idea and a product that people wanted and were willing to pay for but does that mean the validation technique was a good one? I don’t think so.

Here is one of the most well known people in the startup space saying that my idea for Informly was great:

Here are some of the things that people around me said at the time of pulling it all together:

  • “I’m not sure if this email will make it to you, but you’ve managed to build the software most of us wished we already did!” (didn’t end up becoming a paid customer)
  • “Thanks for helping to solve a problem most of us face every day” & “Great work man! I use this product frequently and have recommended it to quite a few people.” (didn’t end up becoming a paid customer)
  • “I have just jumped onto the new platform from http://inform.ly and love it!” (didn’t end up becoming a paid customer)
  • “I’m in love with it. Let me know if you ever need a testimonial. I’ve been waiting for this all my life”. (didn’t end up becoming a paid customer)
  • “Hey Dan! Informly is amazing! What an epic idea. You can manage everything that matters from one place” (didn’t end up becoming a paid customer)

I could go on. I haven’t included any solicited testimonials above, I had lot of friends telling me it was a great idea and giving me great testimonials as well. But those above are just the unsolicited ones.

Coverage in tech press doesn’t work

When I launched the first version of Informly I had it down as a validation goal to get featured by a respected international tech publication. I would have definitely settled for a local Australian one. In the end I was well stoked to get covered by these and more:

This was pretty remarkable given:

  • I’d never built a software product before
  • I had more or less $0 marketing budget
  • I had no network in the startup community
  • I had no funding and no co-founders

According to this post by Noah Kagan I should have at least got 12,000 users from the Mashable coverage. Combine that with the others and I should be onto a winner right? Guess how many paid users were sent by the above traffic sources? Zero.

Targeted Surveys don’t work

content_analytics

Not long after realizing that the idea wasn’t going to fly I decided to ‘pivot’ to a content marketing Analytics app. This time I was determined to make sure I ‘validated’ the idea before working on it for 6 months.

So I built a targeted list of keen content marketers and asked them a bunch of questions in a survey. The purpose of the survey was to ascertain:

  • Were people measuring their content marketing?
  • Would they pay for a tool that enabled them to measure it?

It even included specific questions like ‘Would you pay for this if I built it’. The results are above and they are clear (or at least they were to me at the time):

  • People generally weren’t measuring the important metrics for working out if their content was actually resulting in more business.
  • People would pay for it. 20% definitely, almost 60% possibly. Going from a free to paid conversion rate of .3% on the first version of the product I was very excited about these figures.

After I launched however, I found things to be very different:

  • Most of the people in the beta list didn’t even use the product.
  • None paid for it.
  • After launching it to the public the signup rate was well below the last version of the product (based on on-page conversions).
  • I had 3 people sign up to pay for it. 1 cancelled within a week, the other 2 weren’t actually using it yet.

I should blame the product for some of this as it was the best I could do within 6 weeks and needed improvement but the results were shocking to me at the time. How could they be so different from the survey results?

Well as Steve Jobs said ““people don’t know what they want until you show it to them.” The opposite is also true, people don’t know what they don’t want until they see it. Which brings me to the MVP.

MVP’s are a lot harder than they sound

Ah the much loved ‘MVP’ or Minimum Viable Product. The concept, introduced by Eric Reis with the Lean Startup movement is simple. Do the least amount possible to ‘validate’ the idea. Build a ‘viable’ version of the product and get people using it as opposed to getting it perfect before launching.

I find it quite funny that Eric Reis talks about the startup Path in the book. A company who spent 6 months with their team behind close doors before launching path 2.0. The reality is that a lot of great designers and creators don’t want to release a crappy unfinished product. While it’s appealing to take the ‘vision’ component out of being an entrepreneur it often doesn’t play out that way.

Most ideas won’t sell as half-finished unpolished versions of what they should be. Most of the SAAS entrepreneurs I know are slaving away for months if not years on their idea with very little traction. Of course responding to customer feedback is important but is it even possible to release a week-old product that people pay for?

It might be but it’s certainly a lot harder than it sounds. Services are infinitely easier. A week ago I launched a new service WP Curve, within 1 week it had built up more recurring revenue than a year working on Informly.

Why?

There are a lot of reasons but one is that I could easily release a full version of the product from the customer’s point of view within days. On Saturday I decided to launch a 24 hour developer service for WordPress support and small fixes. By Wednesday I had a website and I have the ability to deliver the service in full (for the first week it was me with a live chat on my mobile phone beside my bed).

I don’t care what the lean startup gurus say, that is not possible with software.

Some things just take time

Nathan Barry

Nathan Barry is a well-connected, skillful entrepreneur who recently tried to build a recurring software business in 6 months. The goal was to get to $5,000 a month. He posted recently after the 6 months was up see this great post.

I recently saw Tim Ferris tweet one of Nathan’s blog posts. He is a very well respected entrepreneur, well connected and very skilled. Checkout the ConvertKit homepage <— legit.

It’s not clear to me how much recurring revenue he actually built since he mentions that he is including some trial people in his calculations. But its’ clearly well below half of the $5,000 goal.

Does that mean it’s a failure? Even if it’s $1,000 it’s more than me and more than most of the other software entrepreneurs I chat with regularly. Building a recurring software business is not easy and some things just take time.

Stop fucking with the product

One of the decisions I made in the last few weeks was to focus Informly back on it’s agency version (which sends simple reports to the clients of web design agencies).

To my delight there is a lean startup term specifically for this decision – The zoom in pivot! In reality though the decision really was to just take a breath and wait.

The irony is that this was the original reason for building the product and this version of the product has existed (in various stages of quality) for a long time – at least 3 years). If I wasn’t so obsessed to find a magical quick fix for the last 12 months perhaps I would have already turned this idea into a valid business.

For the first year I was desperate to find that single thing that took my idea from a dream into the next big thing startup. It didn’t happen. So I chopped and changed a thousand times to try to work out the right ‘formula’. Sometimes though the answer isn’t so obvious. The idea doesn’t go viral. It just takes time.

I’m getting regular new trial customers, a few new paid customers a month (more than the amount that are leaving). It’s no viral startup but it’s growing and over time might become something good or even great. Maybe it just needs a bit of space. And a bit of time.

Spoiler alert on the comments above, this product didn’t work either.