Oh Slack, where have you been all our lives ?
Slack just keeps getting better and better for us. SAGrid is a distributed collaboration, and by design needs collaboration tools. For some years, we got by with various tools - skype, IM, google chat, etc. Whatever works, right ? Except no, that just does not work, or rather, it leads to such overhead in managing input from all these different tools that one ends up with just two options :
Burn out or drop out
That’s not an option for us. We need to keep an infrastructure running after all. Things degenerated over the years into multiple single-channel communications happening between site admins and themselves as well as the coordinator (also, the dude writing this). “So what”, right ? “Whatever works”, right ? Er, no. Why ? Well, simply put, finding out what people and services are doing, and then relaying that information to other people and services is not what I signed up for. That’s a robot’s job, and believe me, I’m so ready for our overlords…
I, for one…
Yes, I welcome the robot overlords. It’s here. Slack is it. I’m down with it. I’m talking lower than a crawling kingsnake down. It’s been just over a month and already things have changed; I haven’t even gotten things configured properly yet, but already I just feel so much more on top of the situation. So much so that I feel this urge to tell the world how great things can be… It’s not just about being able to chat to people in something like realtime, in one place, it’s about having those various channels for different topics. It’s about being able to put emotion and context into what we’re talking about, refer other things and get a holistic view of what’s going on. See, Slack does so much for us than just chat.
Github Slack
The first thing we did was get slack to tell us what was going on with our code. Integrating Github repos to slack was stupidly easy… The payoff is huge : now we can code and have contextual information on who’s committing, pulling, closing, opening issues, etc in our #code channel. Slack also tells us how our code is doing since most of the repos have testing built into them either via our own Jenkins service (get to that in a bit) or via Travis or Circle. So we get told who is doing what, and we get told by a third party the quality of that last commit. Saves a lot of scrutiny later !
Silent death, be gone !
One of the main headaches we’ve been having is unreliability of core services. Stuff just died silently for apparently no reason. What’s that you say ? Of course, we have monitors ! Well, actually just a SAM-NAGIOS box which itself is prone to failure…
Who monitors the monitor ?
So, we need some n-th party to monitor our core services and let us know when things get wonky. Instead of settig up yet another VM, I opted to just deploy Boundary sensors on all of the machines. How ? Well, simple : just wrote a little Ansible role for it and it now gets deployed as part of the bootstrap role. Now, Boundary tells Slack on the #monitoring channel when our core services resource usage rises above a certian threshold. I’m assuming it’ll tell me when they die altogether as well !
We’ve also got our nagios box talking to Slack, which provides service-level information to the team as well.
Ansible Slack
Wouldn’t it be great if robots just did all the communication for us and allowed us to get on with writing code and/or being some other form of awesome ? Yes, actually that would be pretty sweet. So, while poking about, I discovered that Ansible actually already has Slack integration in the form of a module ! Now, my personal crush on Ansible is well noted, but having slack tell the others that I’m testing playbooks on our dev site (and what the outcome was) is going to save me a lot of mails. So, we’ve added a couple of pre- and post-tasks to playbooks that talk to the DevOps channels.
Jenkins Slack
Now for the best part… User engagement. We’re getting to the point where our fancy application porting platform can go into production. It’s a souped-up Jenkins instance which heavily use the various integrations I mentioned above. We don’t want random people just logging in to Jenkins though, we want Jenkins to do all the work in the background and inform relevant actors what’s going on. This, it now does. It goes a little like this :
: “Hey someone sent a pull request to the Application repo”
: “You don’t say ! Let’s see if it’ll build…”
: “Hey everyone ! Jenkins is gonna try build something !”
Yep. we’re in love. Well, at least I am.
Come and hang out with us at https://africa-arabia-roc.slack.com and … just feel better you know ?