In my role over the past number of years, I’ve had many great ideological DevOps conversations with many people from various organizations.  I often explain that to me, DevOps is an approach and a mindset.  Whether it’s implementing a deployment automation tool for the first time, strengthening a continuous integration (CI) implementation, or writing infrastructure as code, my approach is always:

• What are the requirements for success?
• What are the technology barriers that need to be overcome?
• Who is the team that I must work with and what are their strengths and weaknesses?

One question that is very often asked of me is what I think is the biggest challenge in DevOps, which I will address in this blog.

The people within your organization and its culture are the biggest challenge in DevOps.  Please don’t take this negatively!  Usually organizations have very good people working for them.  They are very smart, talented people and they are very good at what they do.  However, we are all creatures of habit.  Every day when I wake up I have 1 cup of coffee with a little milk and sugar.  That’s just what I do every day and when I don’t have my coffee I’ve been known to be grumpy.  For many people, learning a new tool or working a different way doesn’t come naturally.  There is the adage of “if it ain’t broke, don’t fix it.”  People want to do their job well, get it done, and go home to enjoy their family.  New tools and processes tend to make people feel uneasy as it’s something new to learn with new potential for struggle or even failure.  Thus, the question becomes, what does DevOps provide to your organization to help with the challenge of change?

If some people are naturally resistant to change, how are you going to change them (this is a bit of a paradox)? Every person has their strengths and weaknesses. DevOps is about breaking down silos, improving transparency and predictability. DevOps is about removing the mundane and tedious with automation so that your engineers can focus on the new and exciting i.e. implementing powerful change. It is our job as DevOps architects to harness each person’s strengths and use the team to overcome any individual weaknesses.  I believe that every person has the capability to learn and grow, so the real question is whether they have the motivation and are being held accountable for change for the better. If people aren’t being held accountable to implement real, positive change, then chances are that your DevOps transformation will fall short of expectations.

How does DevOps motivate change?  From what I’ve seen, again, the challenge of change can be overcome by finding meaningful motivation for an individual. Upper management is always focused on faster, better, cheaper, but how does this translate into a driving force for an individual?  The cultural shift of DevOps includes empowering your engineers to remove the mundane and tedious from their daily work.  Implementing automation within your organization specifically targets the faster, better, cheaper business results.  For example, it’s not worth an engineer’s time for an application deployment to manually stop a web server, copy files around, toggle the web server configuration, start the web server, and finally ping the web application to see if its running with the new updates.  Those steps are laborious and boring to perform and error prone for a human.  How many people dread the bloated release planning meeting?  Is gathering 10+ people in a room to talk about a specific release for over an hour a valuable use of time?  If instead we were to implement agile best practices with backlog prioritization based upon points, tracked this planning work in a team-based tool, and were able to then forgo the bloated release planning meeting every couple of weeks, would that be a good motivation?  This would empower your product managers to have a real-time, accurate picture of their release plan and they would have real predictability about their product lifecycle.  This is in stark contrast to manually keeping a spreadsheet of green, yellow, red status cells and having to go around to various stakeholders asking for a status update.  DevOps motivates change by targeting objective metrics for faster, better, cheaper.  DevOps also motivates change by targeting subjective metrics for engineers by removing the boring, mind numbing work, which opens up time to do true innovation.

So you, Mr. Trail Blazer, have now fully embraced the ideology of DevOps.  How do you sell the idea of DevOps to management as well as the worker bees around you within your organization?  How do you invoke this cultural shift for improvement?

Be patient!!  DevOps is not a silver bullet.  These changes take time and they focus on iterative improvement.  There is no big bang and suddenly your organization is a utopia.  One of my #1s to being successful with a DevOps initiative is that you must have an executive sponsor.  The worker bees need to understand that the top brass is “kindly” asking them to improve.  Sometimes you will find that top brass needs to provide additional motivation or praise depending upon the circumstances.  The faster, better, cheaper usually resonates well with management.  The other piece that resonates well is the improved predictability and reducing risk.  Outages are extremely painful and can cost millions (just ask the airlines in 2015 and 2016).  Invoking DevOps techniques like deployment automation or infrastructure as code mitigate risks and shorten recovery time if/when an outage does occur.  DevOps techniques will improve the quality of life for worker bees within an organization.  Implement standards and ensure worker bees are adhering to those standards.  For example, developers despise the dreaded code merge, thus they should embrace continuous integration techniques like merging smaller changes more often instead of a big bang merge. Working with the Product Owner to minimize work in progress and limit development to current and release.next only reduces a large number merges all together by refactoring code to remove unnecessary branches. Also applying best practices techniques like build once, then have an automated deploy for any environment.  This streamline builds and deployments with clear visibility of which builds are in the appropriate environments instead of developers wasting time waiting for things like a QA environment refresh.  On the topic of environments, embracing the major advances in cloud automation and infrastructure as code.  Dev/QA teams can no longer afford to wait weeks or sometimes months for a new environment.  Its excruciating in today’s world to even wait a few hours, let alone days or weeks.  I have yet to see a team transform over night, but give a good team enough time and a little direction and they will move mountains.

DevOps incorporates techniques for improved communication, teaming, and transparency.  DevOps is about empowering the individual to invoke real, positive change, by removing waste and streamlining.  It isn’t very difficult to sign up for a GitHub account, download Jenkins and create a simple build job, or even spin up a new server in Amazon Web Services.  There are typically good examples and documentation out there for these things via a Google search or reading documentation.  What isn’t ever straight forward is how to motivate and improve an organization for innovation.  This is the real power of DevOps and the real challenge of DevOps.  And with that, go forth and institute some real, positive change!!

Imagine a large organization with many teams and Jenkins pipeline jobs.  Or imagine one or two people who have to maintain many different Jenkins pipeline jobs within a Jenkins Master.  For anyone who has been responsible for maintaining source code of any scale, the source code 101 tactic of good source code maintainability is the implementation of consistency and reuse.  There are a few mechanisms within Jenkins that we’ll call more intermediate or advanced features, but are relatively simple to implement, i.e. global properties and shared libraries.  In this blog, we’ll give an explanation and a few examples of these features.

Global properties can be found at Jenkins->Manage Jenkins->Configure System under the Global properties section.  They are installed by the Global Variable String Parameter Plugin.  These global properties can be used by every Jenkins job that is tied to a particular master.

These simple string name/value pairs can help a great deal with reuse and maintainability.  In the case of a recent project, we used Sonatype Nexus Repository v3 to store all of our packages built by Jenkins.  The current Sonatype plugin does not support Nexus 3, so we used the Nexus Artifact Uploader Jenkins plugin.  This plugin was easy to use, however, it cannot refer to the Nexus login information as specified in Jenkins->Manage Jenkins->Configure System.  Instead you need to explicitly provide the Nexus repository URL, protocol and credentials within the pipeline stage block.  We definitely don’t want to have to specify the Nexus Repo URL and login credentials (we use a service account for all uploads) in every pipeline job as those parameters should never really change.  Also, if they were to change, we would prefer to make the update only in one place. 

The great thing about pipeline code is that global properties and local variables are used in the same fashion.  In the example pipeline stage code above, NEXUS_SERVER (line 21) and NEXUS_CREDSID (line 19) are global properties defined within the Jenkins Master (the value of NEXUS_CREDSID is actually the identifier of a Jenkins credential).  The other variables (NEXUS_ARTIFACTID, ARTIFACT_FILENAME, etc.) are defined at the job level as these need to be changed for each job.  We use this exact stage block in all of our pipeline jobs to upload any built package to Nexus.  Thus we only need to specify the NexusURL and credentialsID parameters one single time within global properties for all jobs to use.  If for whatever reason the Nexus URL needs to change, or our service account needs to change, we can update these properties in one place to take effect in all jobs.

Jenkins has also made it possible to build your own helper functions through the use of Shared Libraries.  As the number of Jenkins pipelines grow within your organization you’ll probably notice some repeating job patterns.  As usage maturity grows, at some point we mostly stop coding pipeline jobs from scratch.  Rather, we look for an existing job with flows similar to what we need in a new job, we copy that code and modify it accordingly.  As we continually copy code for new jobs, we start to notice that we’re reusing the same code over and over again in many jobs.  Rather than repeating code patterns in multiple locations (does not comply with principle of DRY), take things a step further and define that code pattern as a specific helper function that is managed and maintained in one global location. This is Shared Libraries in a nutshell.

In our project we were building Node.js applications.  We were also using Jenkins to automate scripts to deploy the application to an environment.  Our application included an ecosystem.json file (PM2-Deployment) which defined environment-specific properties (DEV, QA, UAT, PROD) for our application.  For the deployment stages of our pipeline we needed to read the ecosystem.json file in order to find and use the environment specific properties for our deployment stages.  We identified this phase of the pipeline as a common pattern and created a shared library for it, then simply referenced the function in each pipeline job.  Using this approach ensures consistency, correctness, and lowers the overall maintenance burden for this section of code.

Here is a snippet of our ecosystem.json file:

Here is our simple pipeline script with the stage to read our ecosystem.json file.  Note that this script also requires the Pipeline: Groovy Plugin, and the Pipeline Utility Steps Plugin. 

In this example, we read the ecosystem.json file and search through the various apps objects within the ecosystem file for the various environment properties we need.  These properties are saved and used in subsequent pipeline stages when calling our deployment script.  Note that since this is scripted Groovy, if the job is running for the first time and the user running the job does not have appropriate privileges, you might need a Jenkins administrator to approve the running of these scripts (In-process Script Approval).  This is a safeguard within Jenkins to only allow execution of approved, arbitrary code.

Our organization is using good reusability practices when writing our Node.js applications, thus we’ll follow this same pattern with our ecosystem file and application deployment processes.  We’ll have this same stage code repeated in many Jenkinsfile pipeline files.  Wouldn’t it be nice if we could streamline our reuse and not have to copy/paste this code many times in many Jenkins files?

Jenkins Shared Libraries provides you the capability to author and share commonly used helper functions and call those functions within a job.  You can roughly equate a shared library to your own custom Jenkins plugin, but without the formality of the Jenkins plugin architecture.  Shared libraries also come with some security and access benefits.  Since an administrator needs to explicitly add a shared library at a desired context, you don’t need to deal with the clunky In-process Script Approval mechanism mentioned previously.  You can add a shared library globally within a Jenkins master (Manage Jenkins->Configure System->Global Pipeline Libraries).  You can also add a shared library at the project folder level (Configure->Pipeline Libraries).  Adding a shared library globally or at the project level gives you some flexibility to setup specific libraries for specific teams or for everyone using a Jenkins master.  See the Jenkins documentation for more details about shared library structure and adding libraries to Jenkins.

We modified our pipeline example above that reads an ecosystem file and returns desired properties.  We created a shared library with helper functions that, when called with an environment parameter, will return the requested property for our application for the specified environment.  Again, the Jenkins documentation is fairly complete for shared libraries, so take a look for more details.

Some highlights in our new pipeline code:

• Note the included shared library on line 1
• We removed the PREFIX variable from the pipeline and placed it in the shared library files.  Our standard is for all ecosystem.json files to use the same environment prefix, thus we account for this in the shared library instead of in individual pipeline files
• Lines 18 and 19 for finding the TARGET_APPLICATION and APP_PORT are now simplified to a single call of our helper function in our shared library

Our shared library follows the standard structure as defined in the Jenkins Share Library documentation (note that in our case our shared library is in GitHub):

Our two helper functions follow the same pattern.  Here is a screenshot of our getTargetEnvPortFromEcosystem.groovy: 

Since Jenkins pipeline scripts are written in Groovy syntax (Pipeline syntax documentation), it makes sense that shared libraries are written in Groovy and that the pipeline script that we used in the first example can be copied almost verbatim into a Groovy file.

In this blog we introduced global properties and shared libraries in Jenkins.  These features promote reuse and long-term maintainability.  If many pipeline scripts need the same global variable, define that variable as a Jenkins Global Property.  If many pipeline scripts reuse the same script function, put that script in a shared library.  For all companies leveraging Jenkins or CloudBees Jenkins for continuous integration and delivery, the consistency and control provided by global properties and shared libraries is bound to improve your Jenkins experiences and perhaps even your overall quality of life (which may be a little bit of a stretch, but we’re hoping).

Read Part 1
Read Part 2
Read Part 3

In parts 1, 2 and 3 of this blog I discussed what I think are the keys to choosing a DevOps tool that works best for your organization and how to ensure that the tool adoption and subsequent process changes will be successful.  In this final part 4 of my blog I will divulge to you a few vendor secrets…

#4 – An individual vendor is there to make a sale.  Do your homework ahead of time.  Nobody wants this process to take longer than it needs to.

As soon as you contact a vendor their sales person will contact you right back.  Hopefully the sales person will be respectful and reasonable, but they will be persistent in reaching you.  A typical sales cycle goes like this:

1. The vendor **should** start with gathering your requirements (again back to #1).
2. The vendor will do a presentation/demo that explains how their software will meet your requirements.
3. The vendor will then propose a proof of concept (PoC) where they come to you to evaluate their software within your organization.
4. Upon a successful PoC, the vendor will push and push and push to close the proposed deal
5. This is when the procurement process really kicks into high gear. Contracts need to be reviewed and agreed upon.  POs need to be evaluated and signed.
6. The purchase is made, you have access to the software, and now the real fun begins J

As you can imagine, this is a significant amount of time and resources on the part of the vendor AND on your part.  Be respectful and reasonable.  Hopefully the vendor is as well.  If you find a show stopper explain it to the vendor and part ways gracefully.  A funny secret about this business is that it’s a small world and technology is cyclical (case in point look at IBM’s 100 year history with its ups and downs).  Time is one of the most valuable resources that we have, so let’s not waste it.

Finally, I’ll end with a secret that no vendor wants to tell you: “Given enough time, effort and resources, you could make ANY tool work for you”.  No seriously!  Unless you’re considering some mom and pop, brand new, open source, one off, mystical magic wand tool, the odds are that you have a team who is capable of making anything work given enough time, attention, and desire to make it work.  That’s the honest secret that no individual tool vendor will tell you.  So when you finally do make a decision, be happy and confident with that decision and then do everything in your power to ensure that decision results in a successful adoption.

Have you checked 3^rd party research from Gartner, Forrester, Ovum, etc. on the tools you are considering?  The reality is that most DevOps tools have become very mature over the years.  The top 3 to 6 tools that most organizations consider have all hovered around each other in 3^rd party research over the past several years.  Also, the open source community has made strides over the past number of years.  It’s like buying a pickup truck for the purpose of hauling a load of bricks.  Are you going to buy the F150, Silverado, or Ridgeline?  You can get them all for around the same price and they’ll all haul the same amount of bricks.  So how do you choose?

Now I’ve come full circle in this blog.  The best way to decide within your organization is to have all the facts straight (blog part 1), understand what is your process for acquiring a tool (blog part 2), get the right people who will back, support, and lead your decision (blog part 3), and finally with the maturity of the market, if you leverage 1, 2, and 3 you should be confident that you will be successful with your decision.  Good luck!!

Learn More

Are you looking to get the most out of your DevOps transition? Work with us and reach out to one of our specialists at sales@perficient.com and download our DevOps guide for more information.

Read Part 1 
Read Part 2

In part 1 of this 4-part blog, I discussed your tool requirements.  In part 2 of this blog I discussed your procurement process and I mentioned key stakeholders.  Here in part 3 of this blog I elaborate on key stakeholders required to make tool purchases.

DevOps is about breaking down silos, improving collaboration, and working faster, smarter, better, etc.  What’s the one basic thing that this requires?  …CHANGE!  The hardest thing to do within an organization is not spinning up new tools or technology.  It’s not meeting a deadline or even onboarding new infrastructure.  The hardest thing to do within an organization is to invoke change, plain and simple.  Thus, to invoke real change, you need to have leadership from your technical champion and executive sponsor.

#3 Who is your organization’s executive sponsor and technology champion for this new tool?  Who is “that guy” that you need to watch out for?

These days there are many tools/ways to solve a technology problem (insert cliché here).  Who is your senior technical architect who can confidently say that all of the possibilities have been measured and the right decision has been made based upon all of your variables?  This person should be pivotal in writing your tool requirements (see blog part 1).  This person should be instrumental in making your tool decision.  This person should be able to confidently articulate to the organization why the technical choice was made.

Over the years, working at many different organizations under the DevOps umbrella, I’ve seen time and time again that the biggest problem within an organization, aside from plain old change, is “That Guy.”  There is always That Guy (or that group) who has done the development thing or the operations thing FOREVER within an organization.  What they do in their mind works just fine for them.  They might pay you lip service that they are willing to change, but as soon as you turn your back on them they go back to what they know, or worse, they try to undermine what your trying to do.

You need a clearly defined technology champion AND executive sponsor to hold people accountable to adopt new tools and processes.  These stakeholders must be true leaders within an organization.  The more the sponsor and champion are involved, beating the drum, incentivizing and glorifying successful work, the better.  That Guy will kick and scream in the car the whole way to the destination unless mom and dad are there to calm him down with a juice box and give him a high-five when he gets out of the car for being such a good passenger.

To that last point, what is the juice box for your organization?  The reality is that people are motivated by incentives.  If I have no incentive to change and in my mind things are working just fine, then why would I change?  Monetary incentives typically work well, but there are other forms of incentives too.  People also need to be held accountable.  If someone isn’t getting on board with the new change, is the technical champion or executive sponsor holding that person accountable?

Since change is the hardest thing to make happen within an organization, having a strong technical champion and executive sponsor who incentivizes the team and holds the team accountable is a huge key to success.

Learn More

Are you looking to get the most out of your DevOps transition? Work with us and reach out to one of our specialists at sales@perficient.com and download our DevOps guide for more information.

Read part 1 here.

In part 1 of this 4-part blog, I discussed articulating your desired tool requirements.  This is important as you need to ensure that the technology team’s needs are met and that the business understands why it is important to make this purchase.  Here in part 2 of this blog I will discuss understanding the procurement process now that you know what you wish to buy.

#2 – Whether you know it or not, your organization has a procurement process, so figure out that process FIRST!

Typically, there are several key stakeholders within an organization that are required to make a purchase:

1. Executive sponsor – If you don’t have an executive who is willing to step up and fight to make a purchase then chances are that a purchase process won’t get very far. We’ll talk about this in part 3 of this blog series.
2. Technical champion – A more senior technical champion/architect within an organization will typically be required to vet a solution. This person needs to give their blessing from a technical perspective that the solution will meet the needs of the organization.  We’ll talk about this in part 3 of this blog series.
3. Budget person/team – Someone somewhere needs to find the budget to make a purchase. Organizations work differently as far as who controls an organization’s budget (accountant, department director, executive, board, etc.).
4. Procurement lawyers – Software purchases require contracts. Contracts require lawyers for legal reviews.
5. Business sign off person/team – Who is at the top of the chain who needs to sign for a purchase? Is that the department head, CIO, board of directors, etc.?

Nobody like to have to dance at 11pm on the last day of a quarter.  It’s not fun for the vendor and it’s not fun for the client (trust me, I’ve been on both sides of that fence).  Find out at the beginning of the process if your organization has preferred vendors.  If so, it might be much easier to work with those vendors.  A vendor will typically put a termination clause on a PO to expire at the end of a month or a quarter.  Work with the vendor to set a reasonable time expectation (then assume the vendor will do everything/anything to shorten that time J ).

Things to consider:

• What is your organization’s average procurement cycle in terms of process and time?
• Does your organization require a master sales agreement (MSA) and is that MSA in place with the vendor(s) who you are speaking with?
• Can your organization make software purchases from partner resellers? Does your organization have preferred partners that could provide you with additional value with a tool purchase?
• Who are your stakeholder sign-off people and are you including them in the decision-making process?
• Have your legal department check the vendor’s end user license agreement (EULA) at the beginning of the process and get their sign off ASAP. EULAs typically don’t change, so why wait to have your legal department review and sign off on them?

Frankly speaking, procurement cycles within an organization are typically not fun to work through.  Therefore, it’s important to know as much about the process as possible ahead of time.  There will be bumps along the road and the process will typically take longer than desired/expected.  However, if you’re prepared at the beginning hopefully things will go as smooth as can be expected.

Learn More

Are you looking to get the most out of your DevOps transition? Work with us and reach out to one of our specialists at sales@perficient.com and download our DevOps guide for more information.

This is a 4-part blog intended to give you some pointers to think about when choosing DevOps tools.  In the interest of full disclosure and trying to be as non-biased as I can be, I spent around 9 years in software pre-sales with IBM.  One of the main reasons why I joined Perficient is because Perficient partners with pretty much every major tool player in the DevOps industry.  Thus, I am free to help my customers choose the best-fit tool solution that meets their needs.  My loyalty is to my customer as I need to make my customer happy no matter which tool they choose to purchase and implement.

#1 – What are your defined, written down, tool requirements that meet your organization’s business goals?

If you don’t have clearly articulated requirements, how are you going to make an objective decision?  How are you going to ensure an apple to apples comparison of different tools that perform the same job?  If you don’t equate those requirements to your businesses’ objectives how are you going to justify your decision to your business stakeholders in order to get sign-off on a purchase?

Requirement areas to remember:

• Budget – Everyone wants the Rolls-Royce of tools, but can you afford it? Does your organization have use it or lose it budget?  What is your fiscal budget cycle and how far in advance do you need to plan for that cycle?
• Schedule – Most organizations are budget and date driven. Do you have reasonable schedule requirements?
• Security – Those pesky security people always get in the way. Work with them from the beginning so there are no surprises.
• Infrastructure – Who is going to own and be responsible for the installation, configuration, and maintenance of your tool? How does this affect your project requirements and budget?
• Subjective requirements – How are you going to objectively evaluate subjective requirements? This includes things like ease of use, maintainability, ease of onboarding the tool, etc.  These requirements are the hardest to objectively evaluate, but they are extremely important.
• Scalability – Do you expect to have a few people or an army use and maintain this tool? Will this tool be able to grow with your organization as your needs grow?

Once you have your requirements written down for your desired tool, the next question to ask yourself is: “how am I going to get my company to pay for this thing?”.  We’ll talk about that in part 2 of my blog…

Learn More

Are you looking to get the most out of your DevOps transition? Work with us and reach out to one of our specialists at sales@perficient.com and download our DevOps guide for more information.