Is startup validation bullshit?
In 2013 I wrote an article called ‘Is startup validation bullshit?’. It was the most popular post I had written at the time and it led to me writing my first book The 7 Day Startup. Since my blog at the time (WP Curve) has been taken down, I found it on archive.org and re-produced it here. I’ll be referring to it in coming weeks as I start working on another SAAS startup. Jump on my emails to follow the journey. I added a few updated thoughts on Twitter today:
July 9 2013 – Is Startup Validation Bullshit? (WP Curve Blog)
Shortly before starting my software app Informly I read the Lean Startup and had lots of conversations with some very smart software entrepreneurs about validation. I also read lots of posts from people who had âvalidatedâ their ideas and had great results.
The problem is what happens if your results arenât as rosy as those examples?
I feel I can offer a unique perspective here because I launched a product (the first version of inform.ly) last year that ultimately failed. It had over 4,000 free signups and only 15 paid customers ($9 / month). I eventually shut down that version of the product. Here are some of the things I learnt.
(The idea was a dashboard that brought together all of your stats into the one place in a simple way).
Validation doesnât work so well when the answer isnât an obvious YES
The popular stories you hear about startup validation go something like this:
- I created a website with a brief video.
- It went viral.
- I bought a yacht.
Dropbox started from a 3 minute video posted to Hacker News (more here). As they progressed through the beta they had their signup list jump to 75,000 people in 1 day. Hereâs a cool slideshare from their founder Drew Houston.
Dropbox Startup Lessons Learned from gueste94e4c
Well thatâs great if you can get 75,000 optins in 1 day sounds like a good idea but if you donât is it a bad idea? Maybe you just donât have much kudos on Hacker News?
The reality is that most ideas arenât going to go viral. Letâs face it, the chances of you coming up with the next dropbox are fairly minimal. But even if your idea is great will a video showing itâs features go viral? There are a lot of factors at play there I wouldnât be betting on it.
Email optin / beta signup totals donât work
As with dropbox above and numerous other examples, email signups is often considered the key indicator of whether an idea is good. With the first version of Informly I had:
- 1,000 people sign up for the beta (3 months)
- 1,200 people entered their email to be notified of launch (around 7% but that was to the entire site and I was doing a lot of content)
These may seem like small numbers but for someone who had taken 5 years to built up an email list of 2,000 people in my last business I really felt like I was onto a winner.
However as Iâve found with this idea and other versions (more later) there is a very big difference between someone entering their email into a box and someone paying you each month for a product.
People saying itâs a good idea doesnât mean it is
As part of validating Buffer Joel Gascoigne âsimply tweeted the link and asked people what they thought of the ideaâ. His post on validating Buffer indicates that he read a lot into this including:
âAfter a few people used it to give me their email and I got some useful feedback via email and Twitter, I considered it validated. In the words of Eric Ries, I had my first validated learning about customersâ.
Really? A few of your mates said it was a good idea and therefore itâs âvalidatedâ? Obviously it turns out that Buffer was a good idea and a product that people wanted and were willing to pay for but does that mean the validation technique was a good one? I donât think so.
Here is one of the most well known people in the startup space saying that my idea for Informly was great:
Here are some of the things that people around me said at the time of pulling it all together:
- âIâm not sure if this email will make it to you, but youâve managed to build the software most of us wished we already did!â (didnât end up becoming a paid customer)
- âThanks for helping to solve a problem most of us face every dayâ & âGreat work man! I use this product frequently and have recommended it to quite a few people.â (didnât end up becoming a paid customer)
- âI have just jumped onto the new platform from http://inform.ly and love it!â (didnât end up becoming a paid customer)
- âIâm in love with it. Let me know if you ever need a testimonial. Iâve been waiting for this all my lifeâ. (didnât end up becoming a paid customer)
- âHey Dan! Informly is amazing! What an epic idea. You can manage everything that matters from one placeâ (didnât end up becoming a paid customer)
I could go on. I havenât included any solicited testimonials above, I had lot of friends telling me it was a great idea and giving me great testimonials as well. But those above are just the unsolicited ones.
Coverage in tech press doesnât work
When I launched the first version of Informly I had it down as a validation goal to get featured by a respected international tech publication. I would have definitely settled for a local Australian one. In the end I was well stoked to get covered by these and more:
- AntHill – How Informly is bringing big data to small business
- Startup Smart – Gold Coast startup develops an app for all the stats that matter
- Mashable – Can the cloud help you make better decisions?
- Shoestring – One screen, all the stats. Introducing Informly.
- The Next Web – Informly brings an integrated view of Web and business stats to small firms
This was pretty remarkable given:
- Iâd never built a software product before
- I had more or less $0 marketing budget
- I had no network in the startup community
- I had no funding and no co-founders
According to this post by Noah Kagan I should have at least got 12,000 users from the Mashable coverage. Combine that with the others and I should be onto a winner right? Guess how many paid users were sent by the above traffic sources? Zero.
Targeted Surveys donât work

Not long after realizing that the idea wasnât going to fly I decided to âpivotâ to a content marketing Analytics app. This time I was determined to make sure I âvalidatedâ the idea before working on it for 6 months.
So I built a targeted list of keen content marketers and asked them a bunch of questions in a survey. The purpose of the survey was to ascertain:
- Were people measuring their content marketing?
- Would they pay for a tool that enabled them to measure it?
It even included specific questions like âWould you pay for this if I built itâ. The results are above and they are clear (or at least they were to me at the time):
- People generally werenât measuring the important metrics for working out if their content was actually resulting in more business.
- People would pay for it. 20% definitely, almost 60% possibly. Going from a free to paid conversion rate of .3% on the first version of the product I was very excited about these figures.
After I launched however, I found things to be very different:
- Most of the people in the beta list didnât even use the product.
- None paid for it.
- After launching it to the public the signup rate was well below the last version of the product (based on on-page conversions).
- I had 3 people sign up to pay for it. 1 cancelled within a week, the other 2 werenât actually using it yet.
I should blame the product for some of this as it was the best I could do within 6 weeks and needed improvement but the results were shocking to me at the time. How could they be so different from the survey results?
Well as Steve Jobs said ââpeople donât know what they want until you show it to them.â The opposite is also true, people donât know what they donât want until they see it. Which brings me to the MVP.
MVPâs are a lot harder than they sound
Ah the much loved âMVPâ or Minimum Viable Product. The concept, introduced by Eric Reis with the Lean Startup movement is simple. Do the least amount possible to âvalidateâ the idea. Build a âviableâ version of the product and get people using it as opposed to getting it perfect before launching.
I find it quite funny that Eric Reis talks about the startup Path in the book. A company who spent 6 months with their team behind close doors before launching path 2.0. The reality is that a lot of great designers and creators donât want to release a crappy unfinished product. While itâs appealing to take the âvisionâ component out of being an entrepreneur it often doesnât play out that way.
Most ideas wonât sell as half-finished unpolished versions of what they should be. Most of the SAAS entrepreneurs I know are slaving away for months if not years on their idea with very little traction. Of course responding to customer feedback is important but is it even possible to release a week-old product that people pay for?
It might be but itâs certainly a lot harder than it sounds. Services are infinitely easier. A week ago I launched a new service WP Curve, within 1 week it had built up more recurring revenue than a year working on Informly.
Why?
There are a lot of reasons but one is that I could easily release a full version of the product from the customerâs point of view within days. On Saturday I decided to launch a 24 hour developer service for WordPress support and small fixes. By Wednesday I had a website and I have the ability to deliver the service in full (for the first week it was me with a live chat on my mobile phone beside my bed).
I donât care what the lean startup gurus say, that is not possible with software.
Some things just take time
Nathan Barry
Nathan Barry is a well-connected, skillful entrepreneur who recently tried to build a recurring software business in 6 months. The goal was to get to $5,000 a month. He posted recently after the 6 months was up see this great post.
I recently saw Tim Ferris tweet one of Nathanâs blog posts. He is a very well respected entrepreneur, well connected and very skilled. Checkout the ConvertKit homepage <â legit.
Itâs not clear to me how much recurring revenue he actually built since he mentions that he is including some trial people in his calculations. But itsâ clearly well below half of the $5,000 goal.
Does that mean itâs a failure? Even if itâs $1,000 itâs more than me and more than most of the other software entrepreneurs I chat with regularly. Building a recurring software business is not easy and some things just take time.
Stop fucking with the product
One of the decisions I made in the last few weeks was to focus Informly back on itâs agency version (which sends simple reports to the clients of web design agencies).
To my delight there is a lean startup term specifically for this decision â The zoom in pivot! In reality though the decision really was to just take a breath and wait.
The irony is that this was the original reason for building the product and this version of the product has existed (in various stages of quality) for a long time â at least 3 years). If I wasnât so obsessed to find a magical quick fix for the last 12 months perhaps I would have already turned this idea into a valid business.
For the first year I was desperate to find that single thing that took my idea from a dream into the next big thing startup. It didnât happen. So I chopped and changed a thousand times to try to work out the right âformulaâ. Sometimes though the answer isnât so obvious. The idea doesnât go viral. It just takes time.
Iâm getting regular new trial customers, a few new paid customers a month (more than the amount that are leaving). Itâs no viral startup but itâs growing and over time might become something good or even great. Maybe it just needs a bit of space. And a bit of time.
Spoiler alert on the comments above, this product didn’t work either.