Understanding DevOps – Part 4: Continuous Testing and Continuous Monitoring

What is the goal of Continuous Integration? Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that from other developers and with other components of the application, functions and performs as designed. Once the application has been deployed to a production system, it is also a goal to monitor that application to ensure that it functions and performs as designed in a production environment, as it is being used by end-users.

This brings us to the two practices of DevOps that are enabled by Continuous Integration and Continuous Delivery. Namely:

  • Continuous Testing
  • Continuous Monitoring

Continuous Integration and Delivery are both (almost) meaningless without Continuous Test. Not having motoring and hence, not knowing how the application is performing in production makes the whole process of DevOps moot. What good is having an streamlined Continuous Delivery process is the only way you find out about your applications’ functionality or performance being below par is via a ticket opened by a disgruntled user?

Let’s break this down.

Continuous Integration and Continuous Delivery facilitate Continuous Testing

Individual developers work to create code – to fix defects, add new features, enhance features, or to make the code perform faster – could be one of several tasks (work items) they could be working on. When done, they deliver their code and integrate it with the work done by other developers on their team, and with unchanged code their team owns. They would along the way run Unit Tests on their own code. Once the integration is done, they would do Unit Tests on the integrated code. They may run other tests such as White box Security tests, code performance tests, etc. This work would then be delivered to the common Integration area of the team of teams – integrating the work of all the teams working on the project and all the code components that make up the service, application or system being developed. This is the essence of the process of Continuous Integration. What makes this process continuous is where the individual developers’ code is integrated with that of their team as and when they check-in the code and it gets delivered for Integration. (For more detail, read Part 2 of this Understanding DevOps series). The important point to note here is the goal of the Continuous Integration process – to validate that the code integrates at all levels without error and that all tests run by developers run without error. Continuous Testing starts right with the developers.

After validating that the complete application (or service or system) is built without error, the application is delivered to the QA area. This delivering of code from the Dev or development environment to the QA environment is the first major step of Continuous Delivery. There is Continuous Delivery happening as the developers’ deliver their code to their teams’ integration space and to the projects integration space, but this is limited to being within the Dev space. There is no new environment to target. When delivering to QA, we are speaking of a complete transition from one environment to another. QA would have its own environment on which to run its suites of functional and performance tests. DevOps principles preach that this environment be production-like. In addition QA would potentially also need new data sets for each run of the suites of tests it runs. This may be even one or more times every day as Continuous Integration leads to Continuous Delivery at a steady stream. This means that the Continuous Delivery process would not only require the processes to transition the code from Dev to QA, but to also ‘refresh’ or provision new instances of QA’s ‘Production-like’ environment, complete with the right configurations, and associated test data to run the tests against. This makes Continuous Delivery a more complex process than just ftp-ing code over. (I will discuss this in more detail in a later post of Infrastructure automation and Infrastructure as code). The key point to note is that the goal of Continuous Delivery is to get the code ready for test. To get the application to the right environment – continuously, so that it can be tested – continuously.

If one extends the process described above to delivering the service, application or system to a staging and eventually a production environment, the process and goal remains the same. The Ops team wants to run their own set of smoke tests, acceptance tests and system stability tests before they deliver the application to the ‘must-stay-up-at-all-costs’ production environment. That is done using the Staging environment. This is a production-like environment that needs to be provisioned just like the QA Environment. It needs to have the necessary scripts and test data for acceptance and performance tests that Ops would run. Only when this last phase of Continuous Testing is complete would the application be delivered to Production. Continuous Delivery processes would perform these tasks too – provision staging and production environments and deliver the application.

Continuous Monitoring

In Production, the Ops team manages and ensures that the application is performing as desired and the environment is stable via Continuous Monitoring. While the Ops teams have their own tools to monitor their environments and systems, DevOps principles suggest that they also monitor the applications. They need to ensure that the applications are performing at optimal levels – down to levels lower than system monitoring tools would allow. This requires that Ops teams use tools that can monitor application performance and issues. It may also require that they work with Dev to build self-monitoring or analytics gathering capabilities right into the applications being built. This would allow for true end-to-end monitoring – continuously.

The Four Continuous Processes of DevOps:

It would not be much of a stretch to say that the practice of DevOps is really up of the implementation of these four basic processes:

  • Continuous Integration
  • Continuous Delivery
  • Continuous Testing
  • Continuous Monitoring

Of these, the end goal is really Continuous Testing and Continuous Monitoring. Continuous Integration and Continuous Delivery get the code in the right state and in the right environment to enable these two processes. Testing and Monitoring are what will validate that you have built the right application, which functions and performs as designed.

In future posts, I will go into more detail of the implementation of these four processes and the associated practices that enable them.

Related Posts

Understanding DevOps:

Adopting DevOps:

Other DevOps Posts:

, , , ,

  1. Understanding DevOps – Part 2: Continuous Integration and Continuous Delivery « Sanjeev Sharma
  2. Understanding DevOps – Part 1: Defining DevOps « Sanjeev Sharma
  3. Leveraging DevOps in a ‘water-SCRUM-fall’ world – my Post on IBM’s Invisible Thread Blog « Sanjeev Sharma
  4. So, what is ‘Water-SCRUM-fall’? « Sanjeev Sharma
  5. IT Automation Digest for November 5, 2012 | Puppet Labs
  6. Monetate’s 12 step program for continuous deployment « Sanjeev Sharma
  7. Understanding DevOps – Part 5: Infrastructure as Code « Sanjeev Sharma
  8. Understanding DevOps – Part 3: Dev vs. Ops « Sanjeev Sharma
  9. Chef for DevOps – an Introduction | Sanjeev Sharma
  10. Understanding DevOps Series!! | the BPM freak !!
  11. Adopting DevOps – Part II: The Need for Organizational Change | Sanjeev Sharma
  12. Adopting DevOps – Part I: Begin with the Why | Sanjeev Sharma
  13. The State of DevOps (by PuppetLabs) | Sanjeev Sharma
  14. Adopting DevOps – Part III: Aligning the Dev and Ops Teams | Sanjeev Sharma
  15. Slides: IBM Innovate 2013 Session – Continuous Integration for System z | Sanjeev Sharma
  16. Slides: IBM Innovate 2013 Session – DevOps for Mobile Apps | Sanjeev Sharma
  17. Slides: IBM Innovate 2013 Session – DevOps 101 | Sanjeev Sharma
  18. Slides: Mobile to Mainframe – the Challenges of Enterprise DevOps Adoption | Sanjeev Sharma
  19. My ‘meetup’ with Gene Kim | Sanjeev Sharma
  20. Updated DevOps 101 Deck | Sanjeev Sharma
  21. Understanding DevOps – Part 6: Continuous Deployment | Sanjeev Sharma
  22. Adopting DevOps – Part IV: Adopting Continuous Deployment | Sanjeev Sharma
  23. Five areas of documentation that need continuous monitoring « Service Management 360
  24. The 5 enablers of ‘DevOps as a practice’ | This and that
  25. My First Video Blog: Understanding DevOps (A 6 min Overview) | Sanjeev Sharma
  26. IBM Pulse Session: Continuous delivery – accelerated with DevOps | Sanjeev Sharma
  27. Video Blog: DevOps – Where to Start? | Sanjeev Sharma
  28. Video Blog: Continuous Delivery vs. Continuous Deployment | Sanjeev Sharma
  29. DevOps vs Outsourcing | Sanjeev Sharma
  30. What to Expect When Transitioning to DevOps
  31. Six Core Capabilities of a DevOps Practice | The New Stack
  32. Understanding DevOps – Part 7: Continuous Improvement | Sanjeev Sharma
  33. Slides: DevOps and Application Delivery for Hybrid Cloud | Sanjeev Sharma
  34. The 5 enablers of ‘DevOps as a practice’ | This and that
  35. Importance of the Testing Mindset in DevOps | LATEST SOFTWARE TESTING NEWS
  36. 测试理念是实现DevOps不可或缺的重要环节 | 8090Test软件测试网
  37. Adopting DevOps – Part V: Overcoming ‘Cultural Inertia’ | Sanjeev Sharma

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 56 other followers

%d bloggers like this: