This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Wednesday, April 1 • 2:25pm - 3:05pm
Chef Patterns At Bloomberg Scale
Chef is an easy choice for managing a large infrastructure -- at Bloomberg we find it compelling to automate all our things. However, as your infrastructure grows into an interconnected automated organism, software patterns emerge. For Bloomberg's Hadoop clusters, our current set of open-source patterns emerged after only 10's of hosts, running 10's of recipes each. These patterns have been instrumental in building our even larger systems.

This talk will walk through our large scale cluster management patterns:

* Synchronous multi-machine service rolling-restart integrating Chef and Zookeeper for service restart

* Multi-process monitoring service restart (more than subscribes) process table inspection for accidental or human intervention

* Deploy_* providers for Hadoop -- the Hadoop Distributed File System presents unique challenges to Chef for deploying applications and artifacts. We provide primitives such as HDFS file, directory and template amongst Kafka topic creation, HBase table creation and even Kerberos support

* Wrapper cookbook patterns:
- Proxy pattern for pluggable actions (e.g. service restart wrapping) without picking through the run_context
- Pitfalls for pluggable templates

* Virtualized per-cluster Chef servers, built via Jenkins to be generic using Vagrant with the ability to (re)provision using chef-solo and Chef REST API and provide PXE to production operations

avatar for Amit Anand

Amit Anand

Amit Anand has over 17 years of experience in the Information Technology field, as well as extensive experience in big data, including expertise in designing, deploying and managing large Hadoop clusters and eco-system tools. Amit is a certified Hadoop administrator and is currently working on the Bloomberg Hadoop Infrastructure team, where he is managing a large Hadoop installation that includes many production clusters with deployments across... Read More →
avatar for Clay Baenziger

Clay Baenziger

Clay Baenziger - leads the Hadoop Infrastructure team at Bloomberg. Clay comes from a diverse background in systems infrastructure and analytics. At Sun Microsystems, his team built out an automated bare-metal Solaris deployment tool for Solaris engineering labs and his contributions were core to the OpenSolaris Automated Installer. Providing a good introduction to Hadoop, his team at Opera Solutions built out a financial portfolio analytics... Read More →

Biju Nair

Software Engineer, Hortonworks
Biju Nair is a software engineer interested in operating systems, data management systems, system performance and scalability. He has worked on delivering large enterprise applications and software products on multiple platforms. He is a member of Hortonworks Inc and currently working with Bloomberg Big Data R & D team.

Wednesday April 1, 2015 2:25pm - 3:05pm
Mission City B1-B3