DevOps

Configuration Management DevOps Tools are plentiful.  So we will start this primer off with what I look for in a tool and why.  Then we will talk about my current top paid and open source choices.  

When I am evaluating new tools in this class look at the following things:
* Is it OS restricted?
    If I need to manage Windows Machines and it only supports Linux then the solution obviously won't work.
* Does it support the application platform we need to support?
    This is generally more of an Enterprise problem where you are deploying Application Servers.
* Can I write custom scripts?
    If we have a team that can or is willing to learn how to script we can fill in any minor missing pieces.
* How much OS needs to be in place before I can start using it?
    Will the solution allow me to go from Bare Metal(i.e. no Windows or Linux) to fully installed?  Or does it require that we have some basic level of an operating system.
* Is it Agent Based, Agentless, or a Hybrid?
    I tend to lean towards Agentless or Hybrid solutions because it removes the requirement to monitor or restart the agent when they fail.
* Does the tool have a DSL or Domain Specific Language or does it use a standard method for describing work to be done?
    This will tell you how steep an adoption curve you are going to have.  DSL products generally require more training than ones based on YAML or XML.  

This list narrows the field for me.  The how much OS needs to be in place is one most people miss in their lists.  But if you need to roll out machines for something like a Disaster Recovery Drill it can be critical to your timing and person power requirements.  A tool that will go from bare metal to fully functional would be better for that solution.

No solution will be a 100% fit for your environment.  So you need to make trade offs.  If I have a team that can script then customization may not be a problem.  If I don't or if they are all new to it I may want to choose a tool that limits what we can do but does more out of the box.

How the tool works for me is also important.  How do I scale this tool as we grow. Can I set up more than one master server for failover/load balancing?  There is nothing worse than a fully deployed management tool with not management server.

What happens when the Configuration Management tool and server cannot talk for an extended amount of time.  Can the tool maintain the configuration the last known good state?  Does the server alert/log that the server cannot connect to the remote machine?

Once I have figured out a spreadsheet with the names of DevOps tools down one side and my critical items listed across the top.  Below is an example one I filled out for WagsWorld.com(one of my other sites)  If the item isn't a simple yes/no answer I have often used a numeric grading system.  This let's me further refine my rankings of the tools.

If you are a small company working with Linux based systems any of the open sourced tools should be more than enough.   Small companies with Windows systems will have a slightly harder time because that will reduce your Open Source options and may require a small investment in a paid for solution. The larger the organization the more people that have to see how it all works out and the more likely you will choose to pay for a tool.  The good news is that most of them are pretty cheap at this point. 

The bottom line:
* Take your time do your research 
* Make sure it will do at least your list of must haves
* Make sure you understand what it will take to extend it's operations.
* Test it out in a lab before agreeing to spend any money for a tool
* Invest in training if it's available for two or more people to get familiar on the tool

Devops Mastery - Episode 14 Configuration Management - DevOps Tools

 Before we begin this discussion remember this one thing while you read this; what you use to manage code is less important than making sure you manage it.  Management in this context means version tracking, an ability to roll back changes in code, and being able to compare two versions of the code in question.  The time people spend trying to track which is the latest version of a backed up file far exceeds the time it takes to learn any repository management tool.  Wasting time arguing the merits of the tools is just that a waste of time.  While I am not a fan of certain tools in this class they all can handle the basics I have outlined above.  I believe so strongly in this that I am not going to share my personal choice or make a recommendation in this article.

This tool set has been the domain of developers in most organizations since the dawn of computing.  With the focus on software defined infrastructure operation teams need to use these tools. They need to become as adept at using repository management tools as they are with configuring an interface or building out a database.  

The older systems all worked and did the basics with varying degrees of ease and success.  More modern tools like the FLOSS worlds favorite GIT try to advance the tools.  That doesn't mean that GIT doesn't have it's issues or drawbacks.  

These tools generally fall into two sets.  The first is a centrally managed system where a server is required to do the basic commands like checking in code.  The second is a distributed system which allows everyone to manage locally without a server.  This second type then allows you to push the local copy up to a central server for distribution to others.  The modern approach and trend is towards distributed systems.  They give the people writing the code the most control and flexibility over what they are working on by not requiring a network connection to the central server.  I can be completely disconnected from any network and still get all the benefits of the code management system.  If I write a new piece of code on a flight from Cleveland to Dallas I can test it locally and then check it in.  Then when I get a connection back to the central server I can pass the code off to other people and complete integration testing and other tasks.

So how do you decide which one to use?
In most cases people in a company will have experience with a tool set.  It may sound like a cope out but using the one the majority of people are familiar with in this case is the best place to start.  Other things to consider are:
* Cost of the tool - Free, cost per user, cost per X floating/concurrent users
* Operation system requirements of the tool - Is enough of the functionality available to all the OS platforms in your company
* Consistent functionality across operating systems
* Integration with other tools in the environment
* Compatibility with other tools - Support or Plug-in availability with development, Continuous Integration/Deployment systems and testing suites
* How large a learning curve is there for your entire team?

How many different tools in this class should you choose?
Normally a company standard is chosen.  This does not always work.  Most notably if your company is split between Windows and Linux development.  Tools like Microsoft's Team Foundation Server(TFS) work great for .net developers.  Linux developers are left with less than optimal tool sets to interact with the Server.  Microsoft and other tool providers have recognized this issue and implemented ways to use different tools in a single management interface.  In Microsoft's case, this gives people the option to use native TFS and GIT for instance.  This may not work for everyone.  So having more than one option may make sense.  Going beyond two options normally costs exponentially more from an on-boarding and employee training perspective.  So one is optimal but two is tolerable.

Who should own this class of tools?
Everyone in IT.  Ownership is different than management.  Owners have a vested interest in the health of the system they own.  In this case both Developers, Testers, and Operations Teams can be affected by the choices and processes used to do things like upgrades and maintenance.  So everyone should be represented when decisions about the tool(s) are made.

Should you only keep code in the repositories?
Because repositories don't have to be limited to code they make it easy to create work flow procedures based on them.  Often it makes a lot of sense to keep even the requirements documentation in a repository so that everyone can see the history of what and when they changed.  Most of these tools have a method for kicking off other processes when a change has been made to some or all of the repository.  These are called hooks and can do everything from kicking off Continuous Integration/Deployment work flows to sending a message to the next group in the process work flow for review. 

Remember the tool you choose is less important than that you use one.  Discussions about these tools often come with a lot of passion.  So take the time to talk through the options and listen to everyone before making a final decision.  I would also recommend that larger companies choose a small team of people who are passionate about this class of tools to represent the larger groups.  It will  help to manage the process and hopefully build consensus about the final decision.  The people on the team should also agree to be advocates of the chosen tool(s) to the larger group.

Devops Mastery - Episode 16 - Code Management DevOps Tools

What follows are the criteria I use to determine what to script or automate first, second, and so on.  It's another of the "it depends" questions that never have an easy answer.  I will try to help you make some rational decisions but ultimately it's your world to live and you need to decide.

The thing I see everyone want to do first is automate and that takes the longest to do.  That task takes the better part of a week to complete because you are constantly getting pulled in a hundred different directions.  This is not the first thing you need to automate or should automate.  Not that the duration of the task isn't important but the number of times you have to do it, is more important.  I may only spend 2-3 minutes compressing some logs on a development web server.  If I am doing it on 10-20 various servers every week it's not only annoying but it's sucking a lot more time because of the context switching it's causing.  There is also the cost to your company in the time spent by the users of the system trying to troubleshoot the full disk.

So how do I figure out what's sucking up all my time?  Do I track everything I do all the time?  Heck no I don't.  I use a simple spreadsheet to track my work for 3-4 weeks at a time.  I repeat this every 3 months or so.  Then look at what I spent my time on.  I track smaller items with just tick marks and an average time to complete the task.  I don't worry about what time of day it is.  If you want to note the time it takes you for longer tasks it will give you better data but not necessarily better results for this purpose.  Remember the time you are spending on individual tasks isn't as important as the number of times you have to repeat it.   

Now let's flash forward to 3 weeks from now after you have some data collected and answer the following questions.  

    What are the top 10 issues you resolve on a regular basis?
    Are any of the tasks on the list affecting mission critical parts of your organization?  Do you think you should add them to the list? You will likely want to add them and pay close attention to the next item on these tasks.
    How many steps are there to automate in the top 20 tasks you do to resolve these issues? (Exact numbers are not needed, just a really good guess.)

Your top ten list plus the items affecting Mission critical parts of the organization should be your starting point.  To further narrow it down start with the items with the least number of steps.  When doing this though make sure none of the steps are "Install the OS" or "Install Oracle".  Those task steps are not really one step but multi-step tasks on their own.  Now that you have your list ask someone else for input.  Be sure to provide them with the complete list.  I have often described myself and co-workers as being on a mission when they start automating.  This is great, shows focus and determination.  It also can in some cases cause mission blindness.  You want so badly to automate that really hard thing that you miss the quick wins lying at your feet.  Asking someone else to look at your lists will hopefully help you keep from being blind.

Take your own time if you need too complete the list.  As much as the company will benefit from your automation efforts, your work life balance will be the most improved. So spending a little of your time up front will give you more of your back time later.  

If you can reduce a series of tasks you do daily that takes 30 minutes into a single script that runs in 30 seconds over the course of a month you will have 9 more hours to do new scripts.  You will also gain from the lack of typos the script will make on Monday mornings.  If you can take that same script and have it run as a scheduled tasks (think cron or at commands), then you can get back the whole 10 hours to put back into scripting the next thing.  

Remember the goal of automation is to free you from the mundane tasks so that you can work on the more complex and proactive ones.  You will never stop feeling like you are spinning your wheels if you can't get out of the daily repetitive mud.  Automation is the way to push yourself back out onto the proactive route you have been looking to reach.


Episode 18 - Devops Mastery  Choosing what to automate first, second, and so on.

If you are really going to master DevOps you need to master communications.  At least the ability to clearly motivate and pull people into the discussion and the goal you are trying to achieve.  Spending an extra fifteen or twenty minutes writing an E-Mail can save you days and sometimes weeks of discussions, arguments and apologies.  I have a lot of tips about how to be more effective in communicating ideas but today I am going to give you my top six.

What is the goal of your communications?
A goal is essential to forming any effective communications.  If you don't really know what you want to achieve with the message how will anyone else.  Some common goals for me when I was an operations lead were getting help with a Memory Leak, let everyone know about upcoming patches to the middle ware, or getting someone to help figure out why a build or deploy is failing.  Once you know what the goal is you can easily expand on it and form a complete thought and translate it into a communications.

What is your motivation for writing the communications?
Sounds weird I know.  Communicating why you are trying to solve the problem often goes a long way towards getting people's sympathy or at least an understanding about why you care about the problem.  If you try to hide this or omit it you run the risk of pushing people away.  That will also draw people in.  In those cases where you are trying to help someone else it will also draw them into the communications and compel them to read on.  So state it clearly but don't go and on about it.  I normally try to keep my motivations locked up in one sentence.  To much discussion can come off as bragging, whining, or self-righteousness depending on what is being said.

What are you trying to motivate others to do?
This one sounds super obvious but it is really more about how you say it then what is being asked or said.  This is really the sales pitch part of any communications.  I know that for a lot of people this is the most difficult part.  We all normally hate being sold things so the idea of selling things to others can be difficult. The thing that really good sales people know is that the best way to sell is not to be a pushy sales person.  Instead focus on the benefits of the person you are -selling to- asking for help from.  Everyone wants to look good doing what they do best.  Making people feel important by asking them for help in areas you aren't the best in is also helpful.  A little humility will go a long way towards winning people over to your way of working and thinking.

Stay away from certain exclusive words and use more inclusive words.
I only speak English so I am not sure if this part holds true in all languages.  In English though words like I, me, my, us, Department/Group Names, them and you tend to imply an exclusion of the reader or you as the writer.  This exclusion while a minor thing really can put people off and make them restive to your ideas for various reasons.  Some people will think you are blaming them, others may think you are dumping work on them, and still others just may not know you so they will use these words as an excuse to ignore you.  Instead focus on using words or phrases that imply inclusion like we, our, as a team, etc.  Depending on the context these words will have different levels of affect.  They will always be positive it's just that some situations are just naturally inclusive so the effect of this is just less visible.

Leave the door open to Dialog.
Often times the communications I receive are worded as statements of fact and not statements open for discussion.  They seem to be written by people who have already decided what the best approach is and they want me to rubber stamp it.  It's my job as a Solutions Architect and previously as a Team Lead to resist this.  If you are talking about my area of expertise and haven't asked me for input you might even piss me off.  So while stating your plan may be acceptable make sure you are always asking for feedback in the communications.  This could be a requested meeting, a response to the message, or flat out asking what issues the readers see in what you are presenting or proposing.  

Like a good beer let it ferment.
That's right the best thing to do in most cases is just to let the communications sit for a few hours, overnight, or longer if possible.  This is especially good for those times when you are really upset by the actions or lack of actions around a problem.  If the communication is an E-Mail save it and go take a walk.  I will often write an E-mail, save it as a draft, and then go to lunch.  Then when I get back open it up read it I will be fresher and calm.  Normally I re-write about half of it to incorporate the other items on this list.  Even if I wasn't upset when I wrote it I will normally find places where I can make it better.  

Let's look at an example.
As an example here is a less than optimal communication:

"Hey Guys,
I really need you guys to look at this memory leak.  It's effecting the sites performance and the up time stats. When can your team fix it?
Thanks,
DevOps Master"

Something like this would probably work better:
"Hey Guys,
We are all really starting to see the effects the slow performance and a growing number of outages that appears to be caused by a memory leak from the look of the stats.  Let me know when we can get together today to discuss a strategy to identify the real problem and fix it.
Thanks,
DevOps Master"

The second one pulls the reader in, states the problem I need help with, and asks for a meeting to plan the approach.  Remember to always ask for what you want like a meeting today.  Expect that your requests are not always realistic and be accepting of those responses offering a different solution or meeting time.

Never forget that everyone is busy.  The less time everyone spends being irritated with how things are written the more time we all have to fix problems.  So make a effort today to write something better.  Ask others to proof what you write looking for the things in this article.  Look for me to post more tips for improving your communications in future articles.

Episode 19 - Devopsmastery.com -  Six tips for more effectively DevOps communications

Bob from Minneapolis sent us some great feedback and a couple of great questions.  In this post we are going to tackle communicating what scripts are available and what they do.  I spend a lot of my time as a consultant writing up documentation.  At most sites I am pretty sure the work is lost before I get out the door on my last day.  So how did I try to handle this before as a Tech Lead?

To be completely honest it always seems to be hit or miss.  I have never found a single solution that works for all purposes and with all types of people.  Let's face it we as WetWare are the most difficult to communicate with.  Hardware wants power and bits to process.  Software wants data and other inputs.  But humans, a.k.a WetWare, all want something different.  There are all kinds of factors from Gender to skill level to happiness with their current job that can affect and require different types of communications.  Since we can't solve all of these problems, and at least one is unsolvable, we need to focus on what we can.

Let's start with the one thing I know doesn't work and that is network file shares.  It's been my experience that network file shares full of Word or Text documents are where good documentation goes to die.  There is no automatic version tracking, you can never find a structure everyone will agree on, and unless you ingest it into a database somehow it's basically unsearchable.  Since no one can ever find the document they are looking for or be sure it's the most current it quickly becomes a thing we do because Management says we have to not because we find it at all useful.

Wiki's can work well and solve at least a few of these issues but they still aren't a cure all.  They tend to be search-able and track versions.  They still have the structure issue though.  If Google has taught us nothing it is that we don't have to care how things are stored in the great database that is the Internet as long as it can be searched we can find anything.  With that being said the structure is probably less of an issue with a Wiki because they are generally speaking searchable.  So it's always going to stay on my list.  There are so many options from your own version of the Wikipedia to all javascript single page solutions.  It's still not perfect but it works better than a file share and generally looks better too.

Version Control systems like Git believe it or not when combined with something like gitlabs, gitblit, or github(public or private) can also work as a suitable solution. It keeps the documentation with the code and is search-able.  Since the code and documentation are kept together it works a lot better for scripts and programs than it does for something like the procedure to shutdown your whole data center.  It doesn't mean it can't or shouldn't use it just that it may not work great for all use cases.  So again we are close but still not a home run.

Knowledge-base/Content Management Systems/anything like it all work to varying degrees.  This concept is really more implementation and software dependent.  For the most part the difference between these and a Wiki is generally the flexibility to document changes to the documents and who made them.  These systems tend to be very rigid and generally require defining a taxonomy to make them work efficiently.  There is nothing wrong with defining a taxonomy but it normally provides very little reward for the time and frustration put into them to get everyone on the same page.  They generally just become difficult to manage over time and then grow less useful as people stop putting the effort in.

What seems to work in a lot of environments is a blending of the Wiki and Version Control(GIT) concepts.  Yes your documentation ends up somewhat scattered but if everyone knows where to look for which kind of document then it works pretty well.  I generally suggest putting things like procedure and policy items go in a Wiki so they are accessible to everyone.  Then for things that are focused on an applications whether purchased or internally developed being kept in a Version Control system with the code or binaries.  This keeps the specifics of applications which tend to change more often with the related applications.  The wiki can then be used for the things that should be more accessible and slower changing like the procedures for deploying that application.

At the end of the day the best system is the one that wins the popularity contest in your organization or team.  If the whole team hates wiki's then don't use them.  At the same time if it's just the one user who doesn't want to use a wiki then maybe it's time to talk to them about leaving your Company. So go now and document something and put it where you think everyone should find it.  Then ask everyone to send you an e-mail with the first place they looked.  If the majority were looking in the right place then you found your location.

DevOps

user807675373