A Complete DevOps Pipeline Demo Part 1

In work we’ve discussed tools and implementations we want, yet often we talk about each individually. This makes it hard to really evaluate what the fully implemented pipeline would be like to work with.

 

So to give an idea, I’m pulling together multiple docker containers and connecting them to provide a local infrastructure to test with. This is primarily built around the idea of ChatOps which has to primary benefits:

  1. Developers don’t need to login into multiple services, but can orchestrate everything from a single location
  2. The team will gain cohesion as they learn what everyone else is working on, as status updates come in from different services etc. Osmosis!

DevOps

I’m going to walk through several components:

Docker Machine

I’m running this on OSX and so using Virtualbox.

With the plan to run all these containers off a single docker machine, I’m boosting the memory to 8gigs. If this is happening behind a proxy, set the information and set the no_proxy info. Setting the no_proxy info may take a few tries – setup the machine, get the ip via

docker-machine create \
-d virtualbox \
devops;
docker-machine ip devops

For the rest of the article I’ll assume that the IP was 192.168.99.100. Now run the following code with everything together:

docker-machine create \
-d virtualbox \
--virtualbox-memory 8192 \
--engine-env HTTP_PROXY=<proxy host>:<proxy port> \
--engine-env HTTPS_PROXY=<proxy host>:<proxy port> \
--engine-env http_proxy=<proxy host>:<proxy port> \
--engine-env https_proxy=<proxy host>:<proxy port> \
--engine-env no_proxy=192.168.99.100
devops

 

Mattermost

The first step is setting up the central chat server. Otherwise you could always use Slack or HipChat, but since I work behind a proxy, it is easier to have it behind it and call outside.

docker run --name mattermost -d --publish 8065:80 mattermost/platform

Login to the server at http://192.168.99.100:8065 and setup the initial setup.

 

Now go to System Console:

Mattermost System Console Menu Select

Under the Service Settings, you will want to enable Incoming,  Outgoing Webhooks, and Enabling Overriding Usernames from Webhooks:

Mattermost Webhooks Enabled

Click Save at the bottom and switch back to your channel:
Mattermost Channels
Go to your Account settings, we need to setup the webhooks to work with our bot!
Mattermost Account Settings
Switch to the Integration tab, and click Incoming Webhooks, select the channel you want and click Add:
Mattermost Account Incoming Webhook
Right now, I’m setting this for the default channel – in an actual implementation I’d have a channel per project and potentially generic service channels. Note down this URL, we will use it with Hubot.
Now quickly back to the terminal to run:
docker exec mattermost ifconfig
And you should see the IP address of the container on the docker-machine. Now we are going to increment it by one, as the next container we kick off will register it – which will be our chat bot. Assuming mattermost returned 172.17.0.2, the bot will recieve 172.17.0.3.
Now similar to the Incoming Webhooks, go to Outgoing Webhooks and set the channel, a trigger if you want to, and set the callback URL to http://172.17.0.3:8080/hubot/incoming.
Mattermost Outgoing Webhook
Click Add and we are done with Mattermost!

Hubot

Mattermost is an alternative to Slack that you can manage yourself, and built as an alternative, it is made to integrate well with components built for Slack. Hubot is a chat bot that you can develop and thankfully someone created an adapter for Mattermost: https://github.com/renanvicente/hubot-mattermost
I cloned this to a local machine, called the bot Gort, and built a docker image up from it, based on an existing image.
You will want to update this, set MATTERMOST_INCOME_URL to be the Incoming Webhook URL, and get the token from the Outgoing Webhook and set the variable MATTERMOST_TOKEN to it.
############################################################
# Dockerfile file to build Mattermost Bot container
# AUTHOR: John Doyle
############################################################
# Pull base image.
FROM guttertec/nodejs
MAINTAINER John Doyle <john.doyle@fmr.com>
# Install Yo, Hubot
RUN \
npm install -g yo generator-hubot && \
apt-get update && \
apt-get install -y redis-server && \
rm -rf /var/lib/apt/lists/*
# Define default command.
EXPOSE 6379
RUN /etc/init.d/redis-server start
# Mattermost environment variables
ENV MATTERMOST_ENDPOINT /hubot/incoming
ENV MATTERMOST_CHANNEL town-square
ENV MATTERMOST_INCOME_URL http://192.168.99.100:8065/hooks/xzrqe49sbjg1mncerspt9a31qh
ENV MATTERMOST_TOKEN 63irrdrwejdrtxn1w7wesy6nzh
ENV MATTERMOST_HUBOT_USERNAME gort
# Create Hubot
ADD ./Gort/ /Gort
WORKDIR /Gort
CMD /Gort/bin/hubot -a mattermost

Save this file in the parent directory of the bot and call it Dockerfile.

Now we need to build this into an image with the following command (remove the proxy bit if not needed obviously..) :

docker build --build-arg http_proxy=http://<proxy host>:<proxy port> -t mattermostbot .

The first time it will take 15-20 seconds to compile, future changes will be far faster as it only deals with the deltas.

Now that the bot image is built, you can run the bot in a container:

docker run --name chat-bot -d mattermostbot

We can see the containers running:

docker ps

And we can check the logs of the bot:

docker logs chat-bot

Within mattermost, you can now check if the bot is alive:

Screen Shot 2016-01-31 at 7.26.36 PM

I’ll continue this next with the integration of Jira with Mattermost, and integrating Jira with Bitbucket and Jenkins.

Hackathon Idea Sketch for Shoveling my Roof…

Terminator Frozen

The Intel IOT Roadshow is coming to Boston and they are hosting a hackathon and letting folks dig into their developer kit, which looks pretty damn cool!

Well I’m awake and been thinking about some of the past projects that have demoed and what might be fun to attempt myself.

A bit of context also now, in case folks have not been following Boston weather :) It has been a gruesome winter so far – major snow fall and freezing cold has led the city to have many problems. One of these problems tends to be snow build up on the roof. The roof can only take so much weight and a number of cave-ins have occurred over the last few weeks – and if you are lucky to get away with a cave-in, you need to worrry about ice-damns forming and causing water to leak into the house.

So a few nights ago I finally peaked at the roof and felt compelled to shovel it.. very..very slowly.. This is something I would prefer never have to do again!

With that in mind I’m playing around with an idea of a articulated mini shovel system – like a roomba for shoveling snow!

A shape is needed that will allow it to move snow, but be able to deal with the high winds the roof gets. With this in mind, thinking multiple mini shovels that rotate around a pole that, when angled into the wind, provides very little surface area for the wind to catch on, and multiple threads on each segment allow it to have good traction.

Intel Hackathon - Page 1

 

The shovels need to rotate around the central cylinder, shoveling up what was in front/behind, and quickly rotating around so that the snow on the shovel is flung outwards.

 

Intel Hackathon - Page 2

 

Need some sort of ball bearing/socket going on here so that the first rotation can return to the neutral position.

Intel Hackathon - Page 3

Intel Hackathon - Page 4

 

Some of the general ideas of how it would progress – can’t be as aimless as the roomba as it might fall off the roof! Need to spiral around so that it can detect when one segment leaves an edge and is dangling off the side. From there can build up a plan of the roof and how to shovel it.

Intel Hackathon - Page 5

Intel Hackathon - Page 6

Intel Hackathon - Page 7

RDBMS ODS and No-SQL Warehouse?

crazydataOne of the defining differences between RDBMS and No-SQL alternatives is the CAP Theorem. Working on the financial side and the need to have complete confidence in the data has us using RDBMS systems. Big red flags get raised when one system claims that the investment mangers has money on hand to trade when another starts warning you will take the account into overdraft by doing this.

Yet thinking about the issue we could separate the two sets of data into Update-able data and Non-Update-able. Our ODS system could be an RDMS that ensures our data consistency is good throughout the day as holdings are updated and risk analysis is performed. At the same time our data warehouse and reference data could be No-SQL and be locked to updates from the finance users. Only allow our standard reference data system update the data warehouse and we can be confident that the unalterable history will be correct.

Obviously this comes with the drawback of having to access two separate databases and have a middle tier implement the business logic that requires both sets of data. With our own middle tier batch cycle being pretty data intensive this may be something to look into none the less – will see if I can throw together some performance statistics and report back.

DNA v. DB – SCIENCE!

Reading the Wall Street Journal article, Future of Data: Encoded in DNA, I have to admit I am pretty excited! What could the future hold when we don’t keep data locked away on hard disks scattered around a massive server farm but instead it comes in the form or a microscope’s slide.

Data analysis does not involve creating batch scripts to parse massive amounts of data but instead has you in a lab coat adding different substances to the slide to examine the result.

Please every, let us stand and welcome the Jetson family home!