Saturday
Sep242011

Devops Drop 021




Direct download

Follow John Willis on Twitter: @botchagalupe
Follow Damon Edwards on Twitter: @damonedwards 

 

Notes:

Puppet Conf Recap ...

 

Great show.. first class.. venue, food, content...

 

“Operations as a Strategic Weapon”

Damon and I did our combined talks right after Luke’s Keynote.  I thought we rocked.  They will be posting the videos

 

http://www.slideshare.net/botchagalupe/operations-as-a-strategic-weapon

http://www.slideshare.net/botchagalupe/operations-as-a-strategic-weapon-part-2

 

Devops Cafe Roundtable with Luke, Teyo, James and Scott  ..

Basically the management team at Puppet Labs

 

Scott story about joining Puppetlabs... His Loudcloud experience. 

 

Damon killed.  We talked about Service Orchestration, PaaS, culture patterns.. great stuff... We will post the audio on Devops Cafe site and the Video should be up in a week or two...

---------------------------------------------------

 

Puppet Enterprise 2.0

http://puppetlabs.com/puppet/puppet-enterprise/?htb1

 

A lot of new integration with Mcollective and the GUI...

 

New GUI, right out-of-the-box PE 2.0 automatically discovers all resources – packages, hosts, groups, and users.  Uses Mcollective to discover. 

 

Visually Clone Resources To Scale Quickly, Efficiently, and Reliably(From the GUI)

 

With PE 2.0’s new  compliance capability, you now can visually monitor for any unauthorized changes against your desired-state baseline. Can run compliance reports once a day and watch for changing trends...  Give auditors GUI control to see what they need to see...

 

PE 2.0’s new provisioning capability allows you to quickly and easily create new instances of VMware and Amazon EC2.  KInd of like “Knife” with the added bare metal sauce... 

 

---------------------------------------------------

 

“Operating at Scale”

Pedro Canahuati

 

SRE Manager... 

 

Dealing with issues at “SCALE” and I mean scale....

Switched from XEN to LXC to to overhead at scale...

Been using cFengine for years... About to change to Chef or Puppet.. Looking at both. 

All the #devops thing are going on at FB  CD, Agile in operations, collect and store everything.  Like Google, they had to build a lot of their own stuff.  

They build there own TSDB kind of like Opentsdb.  They have built there own monitoring framework, looking framework (they use Scribe).

 

ODS tool the abstracts and visualizes all events (very cool) 

 

I was able to talk to Pedro at the speakers dinner and the following day.  I am a junkie and groupie for guys like this and stuff like .. we talked about CEP and monitoring.  Also about Chef and Puppet.  

 

---------------------------------------------------

 

Beyond the Node: Arkestration with Noah

John Vincent

 

---------------------------------------------------

 

Puppet and Juju, scaling the cloud

Marc Cluet & Adam Gandelman

 

These boys showed up to a gin fight with a knife... 

Slideware of how you can use puppet and Juju together.  I am not a mean guy unless you propose something that you can’t explain in a presentation...

 

Split brain... Needs to be a hackday .. talked to Dan Bodie about this... Interesting...

 

---------------------------------------------------

 

Mårten Mickos

Eucalyptus

CEO Eucalyptus

 

Great presentation... Talked about what the cloud has done to operations.  Also acknowledges cloud needs devops.... 

 

My Zing question ... great answer....

 

We also had some one on one podcasts with the Redhat guys about Openshift and how it works.  

 

Ended up with an interview with Jay Lyman of 451 group... Post on DTO....

 

Oh yeah  on the way to have drinks with Gene KIm I got to get my picture talked with Merle Haggard.  

Wednesday
Sep212011

Devops Drop 020



Direct download

Follow John Willis on Twitter: @botchagalupe
Follow Damon Edwards on Twitter: @damonedwards 

Notes:

Goteborg 2011 - program

http://devopsdays.org/events/2011-goteborg/program/

 

Friday 14 October and Saturday

 

Yours truly doing the keynote...

----------------------------------------------------------------------

 

Announcing Xeround Cloud Database API

http://xeround.com/blog/2011/09/xeround-cloud-database-api

 

Xeround is an elastic, always-on database-as-a-service
for your MySQL applications.

 

AWS, Rackspce and Heroku

 

Benchmarks against an RDS Large at  $0.44 vs the $0.08 standard instance Xeround

----------------------------------------------------------------------

 

How GitHub Uses GitHub to Build GitHub

http://zachholman.com/talk/how-github-uses-github-to-build-github

 

Everyone can push, everyone can deploy 

Master is always deployable

Deploy 10 to 40 times a day

Pull requests are our code review

Master -> Branch -> Pull request -> Master

Pull requests are RAD no meeting, email is your interface, non techs get involved

 

Culture...   Hack days... make things fun... 

 

Hubot, our valiant Campfire bot, has continued to grow in complexity. A tiny list of his (current) capabilities:

 

-unlock the door to our office

-print out a list of the people currently in the office based on their wifi presence

-find an apartment in the area to rent

-deploy GitHub

-say an arbitrary string over the office speakers

-play an audio sample of deadmau5 to everyone through hacked Propane HTML5

-give you a quote from any movie or TV show

-tell you the build status of any git branch

-track and map packages

-SMS any GitHubber from Campfire

-embed a seven day weather forecast

 

----------------------------------------------------------------------

 

PuppetConf as a Service (PCaaS): Sign up for the Free Live Stream

 

http://puppetlabs.com/blog/puppetconf-as-a-service-pcaas-sign-up-for-the-free-live-stream/?utm_campaign=blog&utm_medium=socnet

 

Mårten Mickos

SRE’s from Facebook and Google

John Vincent @lusis Noah dude

Luke of course

Adrian Cole jClouds

Chad Metcalf Cloudera

Jinesh Varia AWS

Mark Hinkle @mrhinkle

 

----------------------------------------------------------------------

 

 

 

 

Puppet Change Management for DevOps

 

What is Puppet?

At Atlassian, we use Puppet extensively with our internal systems, our Hosted products, and our build engineering infrastructure. Here's how we do it in build engineering.

 

Jira, Bamboo,  Greenhopper Rapid Board

 

Bamboo with puppet...

 

----------------------------------------------------------------------

 

IBM Infrastructure as a Service (IaaS) -

 

http://www-935.ibm.com/services/us/igs/cloud-development/?S_CMP=swnews_twitter#tab:fallpromo/#leadspace:default

 

 

From September 12 – November 11, you can provision select virtual machines at the Toronto, Ehningen, Tokyo and Singapore IBM SmartCloud data centers—subject to availability—at no charge. You can access:

 

Virtual machines to run Linux® (Red Hat or Novell SUSE) or Microsoft® Windows® Server 2003/2008

1 block (256 gigabytes) of persistent storage

 

 

----------------------------------------------------------------------

 

 

DataStax gets $11M, fuses NoSQL and Hadoop

http://gigaom.com/cloud/datastax-gets-11m-fuses-nosql-and-hadoop/

 

Brisk, Hadoop based on Cassandra

 

Neo raises $10.6M for Neo4j as graph DBs take off

http://gigaom.com/cloud/neo-raises-10-6m-for-neo4j-as-graph-dbs-take-off/

 

----------------------------------------------------------------------

 

 

Building Scalable Systems: an Asynchronous Approach

http://www.infoq.com/presentations/Building-Scalable-Systems-Asynchronous-Approach

 

Node.js and rabittMQ

Tuesday
Sep202011

DevOps Cafe Episode 19

Love it or hate it, the long form interview is back! 

Guest: Jesse Robbins (Opscode) 

http://www.technologyreview.com/tr35/profile.aspx?TRID=1108

Direct download

Follow John Willis on Twitter: @botchagalupe
Follow Damon Edwards on Twitter: @damonedwards 
Follow Jesse Robbins on Twitter: @jesserobbins 

Notes:

  • Learn more about Jesse's company, Opscode (makers of Chef)
  • John and Damon will be speaking at and doing a live episode of this podcast from PuppetConf in Portland (9/22)

 

Please leave comments or questions below and we'll read them on the show!

Tuesday
Sep202011

Devops Drop 019



Direct download

Follow John Willis on Twitter: @botchagalupe
Follow Damon Edwards on Twitter: @damonedwards 

Notes:

Building data science teams

Data science teams need people with the skills and curiosity to ask the big questions.

http://radar.oreilly.com/2011/09/building-data-science-teams.html

 

People You May Know (PYMK)  LInkedin, Facebook

Netflix and Zynga

Google, Amazon, 

 

A recent report from the McKinsey Global Institute says that by 2018 the U.S. could face a shortage of up to 190,000 workers with analytical skills.

 

http://tech.fortune.cnn.com/2011/09/06/data-scientist-the-hot-new-gig-in-tech/

 

http://www.dataspora.com/2011/09/data-scientists-or-data-composers/

 

 

-------------------------------------------------------------------------

 

New CycleCloud HPC Cluster Is a Triple Threat: 30000 cores, $1279/Hour, & Grill monitoring GUI for Chef

 

http://blog.cyclecomputing.com/2011/09/new-cyclecloud-cluster-is-a-triple-threat-30000-cores-massive-spot-instances-grill-chef-monitoring-g.html

 

We have now launched a cluster 3 times the size of Tanuki, or 30,000 cores, which cost $1279/hour to operate for a Top 5 Pharma. It performed genuine scientific work -- in this case molecular modeling -- and a ton of it. The complexity of this environment did not necessarily scale linearly with the cores.

 

c1.xlarge instances 3,809

cores 30,472

RAM 26.7-TB

AWS Regions 3    ( us-east, us-west, eu-west )

 

 

 Compute Years of Work 10.9 years

 Spot Instances at an average cost of 0.286 USD / instance / hour (0.036 USD / core / hour). Compare that to the 0.68 USD / instance / hour for the same On Demand instance. That’s 57% savings!

 

-------------------------------------------------------------------------

 

What Exactly is Complex Event Processing Today?

http://blog.cloudeventprocessing.com/2011/09/18/what-exactly-is-complex-event-processing-today/

 

Colin Clark...

-------------------------------------------------------------------------

 

https://github.com/nathanmarz/storm/wiki/Rationale 

 

Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, and is a lot of fun to use!

 

The lack of a "Hadoop of realtime" has become the biggest hole in the data processing ecosystem.

 

-------------------------------------------------------------------------

 

 

Building a Devops team

http://agilesysadmin.net/building-a-devops-team 

 

Brian Henerey, from Sony Computer Entertainment Europe.

 

First interview - remote technical test

Instructions...

Ec2 instance .. install Wordpress with a broken Mysql install 

Tomcat log scraping...

Using screen to watch them...

 

Round 2 - Face to face interview

Whiteboard test

Pair programming 

 

 

 

-------------------------------------------------------------------------

 

 

How to Think Like a Computer Scientist

http://greenteapress.com/thinkpython/thinkCSpy/ 

Learning with Python

 

 

-------------------------------------------------------------------------

 

 

Node.js and MongoDB on Ubuntu

http://cloud.ubuntu.com/2011/09/node-js-and-mongodb-on-ubuntu/ 

 

haproxy to catch inbound web traffic and route it to our node.js app cluster

mongodb for app storage

 

With a sample application....

 

-------------------------------------------------------------------------

 

 

First steps with Cloud Foundry on Amazon EC2

 

http://www.cloudsoftcorp.com/blogs/first-steps-with-cloud-foundry-on-amazon-ec2

 

Setting up an IP address and domain name

Making it start the right modules at boot

 

Friday
Sep162011

Devops Drop 018



Direct download

Follow John Willis on Twitter: @botchagalupe
Follow Damon Edwards on Twitter: @damonedwards 

Notes:

NoSQL Benchmark

http://blog.cubrid.org/dev-platform/nosql-benchmarking/

 

Yahoo Cloud Servicing Benchmark

 

Basic operations are Insert, Update, Read, and Scan. There are basic workload sets that combine the basic operations, but new additional workloads can also be created.

 

This article contains tests conducted on the following products and versions.

 

Cassandra-0.7.4

Although Cassandra’s latest version is 0.8.0, we have decided to use the previous version known to be stable. Because when testing with the 0.8 version, the gossip protocol between nodes malfunctioned and the node up/down information was incorrect.

 

HBase-0.90.2 (Hadoop-0.20-append)
The HBase-0.90.2 (Hadoop-0.20-append) was selected because, if not the Hadoop-append version, there may be problems on decreased durability in HDFS.

 

MongoDB-1.8.1

 

Insert, Read Only and Read and Update

 

Insert - Cassandra kills 

Read and update Cassandra beats HBase by a little 

Read Hbase wins of course but only by a little against Cassandra 

Mongo get blow out...

 

Which leads me into .. why I would love to make this event...

 

------------------------------------------------------

 

Using Cassandra, Brisk, and Mahout to Manage Time Series, and Predict Future Events

http://predictingfuture.eventbrite.com/

 

Datastax ... Brisk  a cassandra based Hadoop...

 

 

------------------------------------------------------

 

What is glu?

http://linkedin.github.com/glu/docs/latest/html/index.html

 

glu is a free/open source deployment and monitoring automation platform.

 

a glu agent  is running on each of those nodes

ZooKeeper is used to maintain the live state as reported by the glu agents (blue arrows)

the glu orchestration engine is the heart of the system

 

Glu Script is a Groovy Class with named closures for the actions... (can be groovy or java)

install, configure, start, stop, unconfigure and uninstall

 

The doc is pretty cool .. however, when I started getting into the state machine stuff I had to stop...

 

Orchestration .. Zookeeper to build live state, compare live and desired state.

generate delta 

 

 

------------------------------------------------------

 

NODE.JS AND THE JAVASCRIPT AGE

http://metamarketsgroup.com/blog/node-js-and-the-javascript-age/

 

Three months ago, we decided to tear down the framework we were using for our dashboard, Python’s Django, and rebuild it entirely in server-side JavaScript, using node.js. (If there is ever a time in a start-ups life to remodel parts of your infrastructure, it’s early on, when your range of motion is highest.)

 

This decision was driven by a realization: the LAMP stack is dead. 

 

1991-1999: The HTML Age.

2000-2009: The LAMP Age.

2010-??: The JavaScript Age.

------------------------------------------------------

 

From $0-100million with no sales people. The Atlassian 10 commandments for startups.

http://blog.businessofsoftware.org/2011/09/from-0-100million-with-no-sales-people-the-atlassian-10-commandments-for-startups.html

 

Jira, Confluence 

 

3 ppl to 300 ppl... 

 

Start with two founders..  50/50 

 

Bootstrapping .. first round is 60M

 

-Sell itself, affordable, global, open 

-Use your own product.... Passionately use your own product...

-Measure everything... Capture everything.... even if you can’t analyze 

-Test everything... 5 users free .. raised money for charity 

-ABM...  ... always sponsor the beer at conference.. like Dyninc...

-Send stuff in the mail.. t-shirts... 

-Make everything into a campaign.. Turned hiring into a marketing campaign - .. send only 4 resumes otherwise you are black listed...

-Don’t be afraid to let your first product will fail.. 

 

------------------------------------------------------

 

Devops Dude of the Week....

 

Jordon Sissel

 

FPM and Logstash and now...

 

eventmachine-tail

https://github.com/jordansissel/eventmachine-tail

 

Jordon Sissel.. 

 

This project contains two EventMachine extensions.

First, it adds an event-driven file-following similar to the unix ‘tail -f’
command. For example, you could use it to follow /var/log/messages the same way
tail -f would.

Second, it adds event-driven file patterns allowing you to watch a given file
pattern for new or removed files. For example, you could watch /var/log/*.log
for new/deleted files.

 

For logstash, the log agents were
event-driven using EventMachine. The log agents mainly get their data from
logfiles. To that end, we needed a way to treat log files as a stream.

There’s a ruby gem ‘file-tail’ that implements tailing, but not in an
event-driven way. This makes it hard to use in EventMachine programs like
logstash.

Thus, eventmachine-tail was born.

Further, the usage patterns for logstash required the ability to watch a
directory (or a file pattern) for new log files.

 

rtail -x "*.gz" "/var/log/**/*"

 

-