Agile methodology

In general software development methodology defines 

• Software development life cycle (SDLC) - define phases or stages of development. Common stages of SDLC are: Planning, Analysis, Design, Building or Coding, Testing, Deployment, Operation, Monitor and Maintain
• Development Processes - Define development process map, sub-processes(sequence of activities or tasks), Events that trigger processes, and Outcomes. 
• Artifacts (Deliverables)
• Roles and Responsibilities

Agile software development 

In 2001, a team of 17 leading software practitioners devised and published what they defined as the “Manifesto for Agile Software Development”. It is a document that defines the values and principles of the Agile software development movement as a summary of how they found “better ways of developing software by doing it and helping others to do it” [1]

Although Agile uses iterative development as a basis but advocates a lighter and more people-centric viewpoint than traditional approaches. Agile processes fundamentally incorporate iteration and the continuous feedback that it provides to successively refine and deliver a software system. The ultimate value in Agile development is that it enables teams to deliver value faster, with greater quality and predictability, and greater aptitude to respond to change. 

SCRUM

Scrum is a subset of Agile. It is a lightweight process framework for agile development and the most widely-used one. Scrum is a framework rather than a methodology. It provides clarity of responsibilities through roles, visibility through artifacts, and opportunities for inspection and adaptation through events. Within this structure, scrum is a container for other processes and tools that are appropriate for meeting the specific needs of a team, organization, or product.

• A “process framework” is a particular set of practices that must be followed in order for a process to be consistent with the framework. (For example, the Scrum process framework requires the use of development cycles called Sprints, the XP framework requires pair programming, and so forth.)
• “Lightweight” means that the overhead of the process is kept as small as possible, to maximize the amount of productive time available for getting useful work done.

A Scrum process is distinguished from other agile processes by specific concepts and practices, divided into the three categories of Roles, Artifacts, and Time Boxes. These and other terms used in Scrum are defined below. Scrum is most often used to manage complex software and product development, using iterative and incremental practices. Scrum significantly increases productivity and reduces time to benefits relative to classic “waterfall” processes. Scrum processes enable organizations to adjust smoothly to rapidly-changing requirements and produce a product that meets evolving business goals. An agile Scrum process benefits the organization by helping it to

• Increase the quality of the deliverables
• Cope better with change (and expect the changes)
• Provide better estimates while spending less time creating them
• Be more in control of the project schedule and state

Scrum Process

The process of scrum is simple and circular, with constant and transparent elements of inspection and adaptation. First, a ruthlessly ordered to-do list called a product backlog is created and maintained. Then top-priority items are selected for a fixed, regular time period called a sprint within which the scrum team strives for a predetermined and mutually agreed upon goal.

Scrum Events

The Sprint is a container for all other events. Each event in Scrum is a formal opportunity to inspect and adapt Scrum artifacts. These events are specifically designed to enable the transparency required. Failure to operate any events as prescribed results in lost opportunities to inspect and adapt. Events are used in Scrum to create regularity and to minimize the need for meetings not defined in Scrum. Optimally, all events are held at the same time and place to reduce complexity.

• Sprint - Sprints are the heartbeat of Scrum, where ideas are turned into value. They are fixed length events of one month or less to create consistency. A new Sprint starts immediately after the conclusion of the previous Sprint. All the work necessary to achieve the Product Goal, including Sprint Planning, Daily Scrums, Sprint Review, and Sprint Retrospective, happen within Sprints.
• Sprint planning - Sprint Planning initiates the Sprint by laying out the work to be performed for the Sprint. This resulting plan is created by the collaborative work of the entire Scrum Team. The Product Owner ensures that attendees are prepared to discuss the most important Product Backlog items and how they map to the Product Goal. The Scrum Team may also invite other people to attend Sprint Planning to provide advice
• Daily scrum - The purpose of the Daily Scrum is to inspect progress toward the Sprint Goal and adapt the Sprint Backlog as necessary, adjusting the upcoming planned work. The Daily Scrum is a 15-minute event for the Developers of the Scrum Team. To reduce complexity, it is held at the same time and place every working day of the Sprint. If the Product Owner or Scrum Master are actively working on items in the Sprint Backlog, they participate as Developers.
• Sprint review - The purpose of the Sprint Review is to inspect the outcome of the Sprint and determine future adaptations. The Scrum Team presents the results of their work to key stakeholders and progress toward the Product Goal is discussed. During the event, the Scrum Team and stakeholders review what was accomplished in the Sprint and what has changed in their environment. Based on this information, attendees collaborate on what to do next. The Product Backlog may also be adjusted to meet new opportunities. The Sprint Review is a working session and the Scrum Team should avoid limiting it to a presentation. The Sprint Review is the second to last event of the Sprint and is timeboxed to a maximum of four hours for a one-month Sprint. For shorter Sprints, the event is usually shorter.
• Sprint retrospective - The purpose of the Sprint Retrospective is to plan ways to increase quality and effectiveness. Scrum Team inspects how the last Sprint went with regards to individuals, interactions, processes, tools, and their Definition of Done. Inspected elements often vary with the domain of work. Assumptions that led them astray are identified and their origins explored. The Scrum Team discusses what went well during the Sprint, what problems it encountered, and how those problems were (or were not) solved. The Scrum Team identifies the most helpful changes to improve its effectiveness. The most impactful improvements are addressed as soon as possible. They may even be added to the Sprint Backlog for the next Sprint. The Sprint Retrospective concludes the Sprint. It is timeboxed to a maximum of three hours for a one-month Sprint. For shorter Sprints, the event is usually shorter.

Scrum Roles and Responsibilities

Scrum has three roles that are equal in status yet separate and independent in function. Each role has a defined purpose directly designed to enhance the productivity of the team.

• Product owner - The what and the when. The Product Owner is accountable for maximizing the value of the product resulting from the work of the Scrum Team. How this is done may vary widely across organizations, Scrum Teams, and individuals. The Product Owner is also accountable for effective Product Backlog management, which includes:
  • Developing and explicitly communicating the Product Goal
  • Creating and clearly communicating Product Backlog items
  • Ordering Product Backlog items; and
  • Ensuring that the Product Backlog is transparent, visible, and understood.

• Development team: Developers are the people in the Scrum Team that are committed to creating any aspect of a usable Increment for each Sprint.
  The specific skills needed by the Developers are often broad and will vary with the domain of work. However, the Developers are always accountable for:
  • Creating a plan for the Sprint, the Sprint Backlog
  • Instilling quality by adhering to a Definition of Done;
  • Adapting their plan each day toward the Sprint Goal; and,
  • Holding each other accountable as professionals.
• Scrum Master: The Scrum Master is accountable for establishing Scrum as defined in the Scrum Guide. They do this by helping everyone understand Scrum theory and practice, both within the Scrum Team and the organization. The Scrum Master is accountable for the Scrum Team’s effectiveness. They do this by enabling the Scrum Team to improve its practices, within the Scrum framework. Scrum Masters are true leaders who serve the Scrum Team and the larger organization. The Scrum Master serves the Scrum Team in several ways, including:
  • Coaching the team members in self-management and cross-functionality;
  • Helping the Scrum Team focus on creating high-value Increments that meet the Definition of Done;
  • Causing the removal of impediments to the Scrum Team’s progress; and,
  • Ensuring that all Scrum events take place and are positive, productive, and kept within the timebox.

Scrum Artifacts

Scrum’s artifacts represent work or value. They are designed to maximize the transparency of key information. Thus, everyone inspecting them has the same basis for adaptation. Each artifact contains a commitment to ensure it provides information that enhances transparency and focus against which progress can be measured.

• Product backlog - The Product Backlog is an emergent, ordered list of what is needed to improve the product. It is the single source of work undertaken by the Scrum Team. Product Backlog items that can be Done by the Scrum Team within one Sprint are deemed ready for selection in a Sprint Planning event. They usually acquire this degree of transparency after refining activities. Product Backlog refinement is the act of breaking down and further defining Product Backlog items into smaller more precise items. This is an ongoing activity to add details, such as a description, order, and size. Attributes often vary with the domain of work. The Developers who will be doing the work are responsible for the sizing. The Product Owner may influence the Developers by helping them understand and select trade-offs.
• Sprint backlog - The Sprint Backlog is composed of the Sprint Goal (why), the set of Product Backlog items selected for the Sprint (what), as well as an actionable plan for delivering the Increment (how). The Sprint Backlog is a plan by and for the Developers. It is a highly visible, real-time picture of the work that the Developers plan to accomplish during the Sprint in order to achieve the Sprint Goal. Consequently, the Sprint Backlog is updated throughout the Sprint as more is learned. It should have enough detail that they can inspect their progress in the Daily Scrum.
• Product increment - An Increment is a concrete stepping stone toward the Product Goal. Each Increment is additive to all prior Increments and thoroughly verified, ensuring that all Increments work together. In order to provide value, the Increment must be usable. Multiple Increments may be created within a Sprint. The sum of the Increments is presented at the Sprint Review thus supporting empiricism. However, an Increment may be delivered to stakeholders prior to the end of the Sprint. The Sprint Review should never be considered a gate to releasing value. Work cannot be considered part of an Increment unless it meets the Definition of Done.

DevOps[4]

DevOps is a set of practices that combines software development and IT operations. It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. DevOps is complementary with Agile software development; several DevOps aspects came from the Agile methodology. From an academic perspective, Len Bass, Ingo Weber, and Liming Zhu—three computer science researchers from the CSIRO and the Software Engineering Institute—suggested defining DevOps as "a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality".[5]However, the term is used in multiple contexts. At its most successful, DevOps is a combination of specific practices, culture change, and tools.[6]

The term DevOps was introduced in 2007-2009 by Patrick Debois, Gene Kim, and John Willis, and it represents the combination of Development (Dev) and Operations (Ops). It has given rise to a movement that advocates bringing developers and operations together within teams. This is to be able to deliver added business value to users more quickly and hence be more competitive in the market. DevOps culture is a set of practices that reduce the barriers between developers, who want to innovate and deliver faster, on the one side and, on the other side, operations, who want to guarantee the stability of production systems and the quality of the system changes they
make.
DevOps culture is also the extension of agile processes (Scrum, XP, and so on), which make it possible to reduce delivery times and already involve developers and business teams but are often hindered because of the non-inclusion of Ops in the same teams. The communication and this link between Dev and Ops do, therefore, allow a better follow-up of end-to-end production deployments and more frequent deployments of better quality, saving money for the company.
To facilitate this collaboration and improve communication between Dev and Ops, there are several key elements in the processes to be put in place, as in the following examples:

• More frequent application deployments with integration and continuous delivery (called CI/CD)
• The implementation and automation of unitary and integration tests, with a process focused on Behavior-Driven Design (BDD) or Test-Driven Design (TDD)
• The implementation of a means of collecting feedback from users
• Monitoring applications and infrastructure

The DevOps movement is based on three axes:

• The culture of collaboration: This is the very essence of DevOps—the fact that teams are no longer separated by silos specialization (one team of developers, one team of Ops, one team of testers, and so on), but, on the contrary, these people are brought together by making multidisciplinary teams that have the same objective: to deliver added value to the product as quickly as possible.
• Processes: To expect rapid deployment, these teams must follow development processes from agile methodologies with iterative phases that allow for better functionality quality and rapid feedback. These processes should not only be integrated into the development workflow with continuous integration but also into the deployment workflow with continuous delivery and deployment. The DevOps process is divided into several phases:
  • The planning and prioritization of functionalities
  • Development
  • Continuous integration and delivery
  • Continuous deployment
  • Continuous monitoring 

These phases are carried out cyclically and iteratively throughout the life of the project.

• Tools: The choice of tools and products used by teams is very important in DevOps. Indeed, when teams were separated into Dev and Ops, each team used their specific tools—deployment tools for developers and infrastructure tools for Ops—which further widened communication gaps.

With teams that bring development and operations together, and with this culture of unity, the tools used must be usable and exploitable by all members. 

Developers need to integrate with monitoring tools used by Ops teams to detect performance problems as early as possible and with security tools provided by Ops to protect access to various resources.
Ops, on the other hand, must automate the creation and updating of the infrastructure and integrate the code into a code manager; this is called Infrastructure as Code, but this can only be done in collaboration with developers who know the infrastructure needed for applications. Ops must also be integrated into application release processes and tools. 

"DevOps is the union of people, process, and products to enable continuous delivery of value to our end users."

The benefits of establishing a DevOps culture within an enterprise are as follows:

• Better collaboration and communication in teams, which has a human and social impact within the company.
• Shorter lead times to production, resulting in better performance and end-user satisfaction 
• Reduced infrastructure costs with IaC.
• Significant time saved with iterative cycles that reduce application errors and automation tools that reduce manual tasks, so teams focus more on developing new functionalities with added business value

Implementing CI/CD and continuous deployment

We saw earlier that one of the key DevOps practices is the process of integration and continuous delivery, also called CI/CD. In fact, behind the acronyms of CI/CD, there are three practices:

• Continuous integration (CI)
• Continuous delivery (CD)
• Continuous deployment 

Continuous integration (CI)

In the following definition given by Martin Fowler, there are three key things mentioned, members of a team, integrate, and as quickly as possible:

"Continuous Integration is a software development practice where members of a team integrate their work frequently

... Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible."

That is, CI is an automatic process that allows you to check the completeness of an application's code every time a team member makes a change. This verification must be done as quickly as possible.

Continuous delivery (CD)

Once continuous integration has been successfully completed, the next step is to deploy the application automatically in one or more non-production environments, which is called staging. This process is called continuous delivery (CD). 
CD often starts with an application package prepared by CI, which will be installed according to a list of automated tasks. These tasks can be of any type: unzip, stop and
restart service, copy files, replace configuration, and so on. The execution of functional and acceptance tests can also be performed during the CD process. Unlike CI, the CD aims to test the entire application with all of its dependencies. This is very visible in microservices applications composed of several services and APIs; CI will only test the microservice under development while once deployed in a staging environment, it will be possible to test and validate the entire application as well as the APIs and microservices that it is composed of.

Continuous deployment

Continuous deployment is an extension of CD, but this time, with a process that automates the entire CI/CD pipeline from the moment the developer commits their code to
deployment in production through all of the verification steps. This practice is rarely implemented in enterprises because it requires a wide coverage of tests (unit, functional, integration, performance, and so on) for the application, and the successful execution of these tests is sufficient to validate the proper functioning of the application with all of these dependencies, but also automated deployment to a production environment without any approval action. The continuous deployment process must also take into account all of the steps to restore the application in the event of a production problem. Continuous deployment can be implemented with the use and implementation of feature
toggle techniques (or feature flags), which involves encapsulating the application's functionalities in features and activating its features on-demand, directly in production,
without having to redeploy the code of the application. 

Tools

We see DevOps culture in CI very clearly, with the spirit of collaboration and communication, because the execution of CI impacts all members in terms of work methodology and therefore collaboration; moreover, CI requires the implementation of processes (branch, commit, pull request, code review, and so on) with automation that is done with tools adapted to the whole team (Git, Jenkins, Azure DevOps, and so on). And finally, CI must run quickly to collect feedback on code integration as soon as possible and
hence being able to deliver new features more quickly to users. 

References

---

1. Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M.,…Thomas, D. (2001). Manifesto for agile software development.
2. Ken Schwaber; Jeff Sutherland. "The Scrum Guide". Scrum.org. Retrieved October 27, 2017.
3. What is Agile. What is Scrum
4. What is DevOps