CTO School Session 3: Peter Bell

• automated unit/acceptance tests
• also biz metrics tests
  • if orders/day decreases significantly, rollback
• perf tests
  • after deploying, look to see if common user behviors are taking longer than normal. if so, rollback

• when the pain of not doing it exceeds the pain of adding new features
• "probably not before you hit product market fit"

• ex: pull in info about twitter friends/followers

  perfect fit for async msg queue

  however, start by doing it sync -- takes a few seconds, but no real pain point there until more load, so don't change it yet

  "what is the simplest thing that will work"

  if you put in a hacky solution, put enough tests around it to avoid breaking the whole system (and reduce the fear of refctoring)

1/2 the room is using user stories (~ 15-20 people)

As a 'role' I want 'feature' so that 'benefit'

if you can't come up with a reason for a user to use that feature, they are probably not going to use it (don't build it!)

INVEST

• I-independent

  N-negotiable (if you can't fit all the info on a 4x6 card, get a bigger sharpie and a smaller card (you are doing too much)

  V-valuable

  • if it doesn't have a benefit, don't do it.
  • If you can't clearly articulate the benefit, don't do it
  • two types of feature ideas
    • one type you think of offhand and then forget about it
    • the other type bugs you every time you use the system, everyone clamors for it
      • those are the ones you should build

  E-estimable

  • needs to be clear that we can figure out how to do it
  • if it's not estimable, maybe it's a spike

  S-small

  • break things down...good size of story is 'half a day'

  T-testable

• not necessarily something releasable
• ex: registration system

  • hardcore--lots of steps, health care, etc etc

    start with the simplest thing (first/last/email/password) now I know who you are

    so simple it won't even validate these things...just laying out a form with a message. (first story)

    • now it's something a biz user can test! "yes, I like that message" or "no, that message is wrong" etc

    now going to require validation (next story)

    • automated test: make it ID-based (look for #registrationError). Don't make it check for a string...too brittle, that string may change

    now add one more screen with some healthcare info (next story)

    at each point ask, "is this an MVP," could we release here?

    this way new stuff is available for internal biz users to view and test on a daily or near-daily basis

• 'happy path'
• add validation
• add data
• add robustness
• ex: algorithm:
  • what's the simplest algo to write that can do something useful
    • something that just returns a 0 or 1, maybe, rather than the full prediction value
  • then what's the simplest improvement you could make?
  • repeat

institutional memory -- how do we make sure we keep the deep knowledge behind something available

• the problem is, most institutional memory has alzheimers...
• the info is on a wiki
• but the wiki gets out of date
• our institutional memory becomes misleading

what acceptance test do you recommend?

• they use rspec

  committed the pretty-print rspec html output and made it available for everyone in the company to see, so they know what's up

+ - when you are small, getting all stakeholders to sit down and agree on everything itself becomes an overhead...?

• yes, all process is an overhead
• ex: daily standup: what did I do yesterday, what am I doing today, what's blocking me?
• you can use an event to drive process
  • let's get beers tonight and make a decision about the specs

can you give ideas for product roadmapping?

• i like eric ries' definition of a lean startup
  • a human institution developing products or services under conditions of extreme uncertainty
    • intuit even decided they were a lean startup
• how much uncertainty do you have?

  • if your customers are signing up for v17 of your software and they're expecting a roadmap and you're expecting them to stick around for a long time, maybe they need it

does a roadmap have to be a fixed direction, or can it be a set point? ( a vision)

• assumption: no one is creating a blind startup

  • == a startup that makes a 2-year roadmap, follows it no matter what anyone says, and checks to see if they are relevant at the end of 2 years

  what's the value of having a roadmap, knowing that you are in an uncertain world, knowing thigns will change?

  diff b/w vision and roadmp.

  • vision: "we are going to staten island"
  • roadmap: "this is how we are going to get there"

reality: VCs are paying for our dev time...THEY want to know what we'll be building for next 3 months

comment: any planning past 6 months goes nowhere...we plan for 3 months (and can usually hit it), but plans for 6-months+ never happen. and these are guys that manage 500+person orgs with millions of dollars behind them. And they had predictable businesses with established pipelines

• 1-week iterations are awesome if you can keep the pace up
• the world is moving toward 2-week iterations
• month is really long these days

• you have some kind of backlog (stuff to do "next")
  • this comes out of manufacturing
• you focus on limiting work in progress

  • goal is to minimize cycle time

    in manufacturing, thngs half-built are worth nothing (you don't get paid until you ship *completed* cars)

    same with programming...all half-finished features add 0 value to biz...kanban says: let's get the NEXT feature done as fast as possible, and get it out

• when an organization has little structure, iterations can be good
  • because they force you to focus for a few weeks
• kanban can help you, too, because it makes you focus on just one thing
• books
  • lean software development (from 2003)

• cost go/no-go (if this takes 3 weeks to build, we can't do it.
• market window
  • if it takes 6 weeks but it's necessary for somethign happening in 3 weeks, don't build it)
• ROI comparison
• not just "because" -- use estimation if it helps you

• most of the time it matters how much or how long it will take
• sometimes it doesn't matter how much it costs (within reason)
• worth knowing which is most important (can spend more to dev fast, versus dev slow but save money)
• what's successful -- 5% over time budget but 1000s use it, or 5% early but no one uses it?
  • if you need estimates granularly at wildly accurate numbers, something is not right
• ideal days:
  • problem with using ideal days is not every day is ideal (meetings, installing software, etc etc)
• story points

  • 1,2,5,epic

    • split epic into smaller amounts

    people always ask "how many hours is a point"? so use ideal days when you're just starting

    to get over the hump, just use ideal days and drop the days. 3 ideal days = 3 points

    don't allow 0-point cards

    • because you can end up with 100 0-point tickets and everyone expects it to happen immediately
    • roll up multiple small ones into a 1-point story

    problem with using larger numbers (10,20,50) is people will end up saying "oh, 47.5"...no one can be that precise

    • small story points numbers makes people not estimate so granularly

    t-shirt sizes (small, medium or large)

• unit
• integration
• acceptance

• you don't improve quality by doing more inspection
  • best way to improve software quality is not to do more QA
    • although you do still need that
• two important things:
  • build in quality by writing automated acceptance tests
  • do qa before you write the code
    • Test-driven development
• if you have a QA resource (tester, manager, whomever)
  • bring them into the meeting when you spec the feature...ask them how they would break/test it
  • write down what they say...that's one of your automated acceptance tests

• very first thing: make sure the developers know *how* to write tests (get a consultant, figure it out, whatever you need to do)

  every time you get a bug, WRITE A FAILING TEST

  • you will start getting clusters of tests in the important parts of your app

  every time you tweak a method, write a test or two

  • the boyscout method...leave the code cleaner than when you got there

• you might have code you know you're going to throw away in a week

  write tests for the parts that scare you, that you don't know well how they work or think they'll break

  write tests for the critical parts of your codebase