ChefConf

Chef is an easy choice for managing a large infrastructure -- at Bloomberg we find it compelling to automate all our things. However, as your infrastructure grows into an interconnected automated organism, software patterns emerge. For Bloomberg's Hadoop clusters, our current set of open-source patterns emerged after only 10's of hosts, running 10's of recipes each. These patterns have been instrumental in building our even larger systems.

This talk will walk through our large scale cluster management patterns:

* Synchronous multi-machine service rolling-restart integrating Chef and Zookeeper for service restart

* Multi-process monitoring service restart (more than subscribes) process table inspection for accidental or human intervention

* Deploy_* providers for Hadoop -- the Hadoop Distributed File System presents unique challenges to Chef for deploying applications and artifacts. We provide primitives such as HDFS file, directory and template amongst Kafka topic creation, HBase table creation and even Kerberos support

* Wrapper cookbook patterns:
- Proxy pattern for pluggable actions (e.g. service restart wrapping) without picking through the run_context
- Pitfalls for pluggable templates

* Virtualized per-cluster Chef servers, built via Jenkins to be generic using Vagrant with the ability to (re)provision using chef-solo and Chef REST API and provide PXE to production operations