The Seven Stages of Bosh

Surviving successful Bosh adoption

Chris Hedley of CloudCredo

Daniel Jones of EngineerBetter

Stage 0: Disbelief & Shock

“the survivor refuses to accept that something is true or real”

What Bosh does for you

  1. Create all the VMs
  2. Auto-healing
  3. Rolling upgrades

Learning Points

  1. Bosh is very powerful and does amazing things
  2. Your team needs to learn a lot

Stage 1: Denial

“the survivor imagines a false, preferable reality”

 

Learning Points

Remember that Bosh gives you:

  1. Declaration of intent
  2. Effectively immutable
  3. IaaS agnostic
  4. Highly automated
  5. Self-healing
  6. Open source

Stage 2: Anger

“the survivor becomes frustrated”
“ Confusion leads to anger.
Anger leads to hate.
Hate leads to production downtime. ”

Yoda - The Phantom Manifest

Bosh is not a power drill manufacturer

Bosh is not a basketball player

Bosh is not BOSH

Bosh MicroBosh Bosh Micro? bosh‑init Full Bosh? Binary Bosh
bosh‑lite light bosh Nise Bosh bosh‑solo
bosh‑gen bosh‑bootstrap traveling‑bosh

JOBS

Jobs are made up of templates

One job's templates are in /var/vcap/jobs

SPECS

packaging spec != job spec != rspec

That's annoying…


$ bosh target

[WARNING] Loading the cli took 12.4 seconds, consider cleaning your 
gem environment

Wait… What?


$ bosh ssh
…
Starting interactive shell on job ha_proxy_z1/0
No RSA host key is known for 10.244.0.34 and you have requested strict
checking.
Host key verification failed.

Do not trust Bosh


Started prepare for update
Started prepare for update > Delete stemcell. Done (00:00:26)

Started deploy micro bosh
Started deploy micro bosh > Unpacking stemcell. Done (00:00:00)
Started deploy micro bosh > Uploading stemcell. Done (00:00:00)
Started deploy micro bosh > Creating VM from ami-e1577396 light. Done (00:00:33)
Started deploy micro bosh > Waiting for the agent

Some time later...

Do not trust Bosh


Started prepare for update
Started prepare for update > Delete stemcell. Done (00:00:26)

Started deploy micro bosh
Started deploy micro bosh > Unpacking stemcell. Done (00:00:00)
Started deploy micro bosh > Uploading stemcell. Done (00:00:00)
Started deploy micro bosh > Creating VM from ami-e1577396 light. Done (00:00:33)
Started deploy micro bosh > Waiting for the agent

Later still...

Do not trust Bosh


Started prepare for update
Started prepare for update > Delete stemcell. Done (00:00:26)

Started deploy micro bosh
Started deploy micro bosh > Unpacking stemcell. Done (00:00:00)
Started deploy micro bosh > Uploading stemcell. Done (00:00:00)
Started deploy micro bosh > Creating VM from ami-e1577396 light. Done (00:00:33)
Started deploy micro bosh > Waiting for the agent
log writing failed. can't be called from trap context
/Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/buffered_io.rb:65:in `recv': Connection reset by peer - recvfrom(2) (Errno::ECONNRESET)
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/buffered_io.rb:65:in `fill'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/transport/packet_stream.rb:104:in `block in next_packet'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/transport/packet_stream.rb:95:in `loop'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/transport/packet_stream.rb:95:in `next_packet'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/transport/session.rb:184:in `block in poll_message'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/transport/session.rb:179:in `loop'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/transport/session.rb:179:in `poll_message'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/transport/session.rb:164:in `next_message'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/session.rb:95:in `block in next_message'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/session.rb:94:in `loop'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/session.rb:94:in `next_message'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/methods/publickey.rb:53:in `authenticate_with'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/methods/publickey.rb:20:in `block in authenticate'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/key_manager.rb:122:in `block in each_identity'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/key_manager.rb:119:in `each'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/key_manager.rb:119:in `each_identity'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/session.rb:79:in `block in authenticate'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/session.rb:66:in `each'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh/authentication/session.rb:66:in `authenticate'
from /Users/deejay/.gem/ruby/2.1.3/gems/net-ssh-2.10.0.beta1/lib/net/ssh.rb:226:in `start'

Of course I'm sure


$ bosh deploy
…
Deploying
---------
Are you sure you want to deploy? (type 'yes' to continue): y
Deployment canceled
Deployment canceled

via @xoebus

Learning Points

  1. Be prepared to work to build a mental model of Bosh
  2. Expect bugs
  3. You'll need to look in many areas

Stage 3: Bargaining

“the survivor seeks compromise”

Bosh has its opinions

Learning Points

  1. Do not argue with Bosh

Stage 4: Guilt

“the survivor believes to have committed sin”

Leaving a Bosh bomb

Learning Points

  1. Same as any niche technology choice
  2. Radiate reasons for using it
  3. Pair on release development

Stage 5: Depression

“the survivor has feelings of severe despondency and hopelessness”

Waiting

Testing

Composition

Extensibility

Job Modularity

Learning Points

  1. Tooling is not great, do not expect it
  2. All or nothing
  3. Use bosh-lite
  4. Consider a pool of Boshes for CI

Stage 6: Acceptance

“a calm, retrospective view for the survivor, and a stable set of emotions”

Learning Points

You now have:

  1. IaaC - not just provisioning
  2. A team capable of end-to-end continuous delivery

Stage 7: Hope

“the survivor develops a positive attitude about the future”

Releases, stemcells, tracker link and documentation all in one place

Open

pivotaltracker.com/n/projects/956238

Vendors will catch up

Bosh Self-Help Guide

  • It won't be easy, but it will be worth it
  • Be willing to become poly-skilled
  • Exploit bosh-lite for acceptance testing
  • Try to avoid Windows

Thanks.

Questions?

cloudcredo.com

@ChristHedley

engineerbetter.com

@EngineerBetter