Making the Change to Agile: How A-10 OFP Development Accomplished the Impossible

Abstract

Supporting a 45-year-old airframe with legacy software written in several languages across decades of engineering efforts is difficult in and of itself. Changing the development lifecycle model to support modern software development business practices is impossible: almost. This paper explores how the A-10 Operational Flight Program (OFP) software development team studied, transitioned, and then embraced agile methods and the results that followed.

Introduction

The A-10 Operational Flight Program (OFP) software development is maintained by the 309th Software Engineering Group (309 SWEG) at Hill AFB, UT. The portfolio maintained by 309 SWEG is comprised of four Line Replaceable Unit (LRU) onboard computers with multiple other hardware projects maintained by various contractors that are tested at Hill AFB with the assistance and support of the 309th. For decades, software development was accomplished using standard waterfall lifecycle management methodologies and practices. In the late 2010’s, the Department of Defense (DoD) began shifting software development practices to agile methods and processes, prompting many programs to begin developing software in ways never attempted on a large scale in the DoD. In 2019, the A-10 OFP software development team took a chance on agile and began investigating how to transition to agile and prove whether it could work.

Development Lifecycles

Waterfall Lifecycle Management for A-10 Software

A Suite release for A-10 OFP using waterfall could take anywhere from three to four years to get from an Avionics System Requirements Review Council (ASRRC) requirement to a released software upgrade.

Figure 1. OFP Schedule Overlap

One issue with this approach was that a requirement given three to four years prior was often irrelevant to the needs of the pilots at the time of the software release, as requirements for the current fight had changed by the time warfighters saw the updates. Comprehensive schedules and plans were put in place for each suite release with two or three suite efforts overlapping each other based on their development cycle. Figure 1 above shows what the schedule overlays for Suites 10, 11, and 12 looked like when Suite 12 was first conceptualized.

This model for software development is slow, risk averse, and subject to massive delays due to any kind of requirements changes. To start a suite release, the pilot community gathered at ASRRC and determined the next set of software requirements. Those requirements were evaluated to determine feasibility, cost, and effort required to complete. A final requirements list, supported by development estimates, was determined, and set for the next suite release. The OFP’s system design team would generate requirements documents that provided detailed explanations and the required architecture changes for implementing each candidate. During that time, the software development team was working on the software updates from the previous suite. As the development team was wrapping up the update for the prior suite, the design team was passing the requirements documents to the development team for the next suite release. The development team would receive the requirements documents and plan execution efforts. The test team would also take the requirements and begin developing test procedures for the software updates. After the development team finished coding, they would pass the code off to the testing team to execute integration, safety of flight, and regression testing. This would happen three to four times during the development cycle. Pilots would test each release to find defects or suggest refinement changes to the candidates. While the development team was working on the code updates, the design team was moving on to the next set of candidates for the next release to keep the cycle going.

After the 309th took over OFP development from Lockheed Martin, the 309th lifecycle schedule became the overall program lifecycle schedule that the A-10 System Program Office (SPO) followed. This caused issues as the overlapping nature of the suite efforts saw milestones happening at different times that didn’t line up with any master program schedule. The 309 SWEG program director was constantly in meetings with SPO leadership and pilots reporting cost and schedule progress and answering questions of why development was falling behind when the pilots kept asking for changes to the requirements.

The traditional waterfall acquisition lifecycle is very big and complex. As shown in Figure 2 below, there are many elements and timed phases that occur throughout a program’s existence. Software should only be one of those inputs to the lifecycle, not the portion controlling the entire schedule. It took until the end of the Suite 9 software release to decouple the OFP lifecycle from the overall A-10 program lifecycle.

Figure 2. DAU Acquisition Lifecycle Wall Chart f[1]

Agile Investigation

In 2017, the A-10 SPO had started an effort to replace several flight panel instruments in the cockpit with a single digital screen they dubbed the High-Resolution Display System (HRDS). As an experiment, the A-10 development team program director asked the HRDS support team to try something they had never tried before: manage the effort as an agile program. The team lead researched what agile was and how to do development using agile principles and practices. The program management team continued studying different agile methodologies with Scrum and Scaled Agile Framework (SAFe®) becoming standouts as possible full implementation options. When the overall A-10 OFP program began investigating agile in earnest in late 2018/early 2019, SAFe® was what large organizations were initially steered towards.

Later that year, the Office of the Chief Information Officer, Mr. Thomas Lam, Acting Director, in collaboration with Mr. Nicolas Chaillan, Special Advisor for Cloud Security and DevSecOps from the Office of the Undersecretary of Acquisition and Sustainment (A&S), released a document entitled, “DoD Enterprise DevSecOps Reference Design” [2] that outlined the merits of DevSecOps and what it means to develop software in a DevSecOps environment. This document urged DoD programs to begin transitioning software development business practices from the traditional waterfall lifecycle management to DevSecOps and made a point of steering programs away from SAFe®.

In early 2019, program management came across the book, Scrum: The Art of Doing Twice the Work in Half the Time, by Jeff Sutherland [3]. This book outlines the benefits of Scrum, how it works, how it changed the way the software industry does business, and how to implement agile. The deputy program director began studying Scrum and teaching the merits to the current A-10 OFP leadership team. The HRDS team was demonstrating Scrum principles and had been for over two years at this point and many of their processes and procedures were copied and used to stand up a trial scrum team for supporting legacy software development.

Agile Test

In October 2019, the first OFP Scrum team was assembled using team members who had expressed little or no interest in trying agile. Team members were selected that way on purpose to give them the opportunity to show it really wouldn’t work and the A-10 program should not adopt agile. The only expectation management levied on this team was they had to follow the tenets of agile and scrum to the letter with minor tweaks to ensure code integrity when integrated into the main development stream later. The Agile Team (A-Team) accepted the challenge and began working in a scrum agile fashion. Management took a step back and gave the A-Team as much autonomy and freedom as they could so as not to interfere with or influence the experiment.

The A-Team was a mashup of expertise and roles: a systems engineer, two developers and two testers. A scrum master was trained and appointed to “lead” the team. A product owner was trained and appointed to ensure development integrity. These team members were taken from their standard teams and moved to an area where they could work and collaborate as a small, independent team.

About halfway through the three-month experiment, the team was surveyed for how things were going. The biggest critics of the experiment were touting the biggest successes they had observed. Different ways of development were being done where the systems and test engineers were right there with the developers helping each other work through the requirements, development, test procedures, and execution. The systems engineers were seeing how their requirements were being implemented and changed the way they wrote and developed the requirements to better suit the needs of the coders. The system engineer and the coders assisted with test procedure development and execution - something they didn’t always participate in in the past.

The combination of expertise led to the development of processes and procedures that made it more efficient for teams to complete their normal tasking. The A-Team took a single candidate and performed a “focus-to-finish” development approach that saw them complete the candidate from start to finish without being diverted to other tasks or candidates, something that happened often. At the end of the three-month trial, the A-Team was asked for a recommendation on whether to extend scrum to the full A-10 OFP development team or return to their normal business practices under waterfall. The recommendation was unanimous to move to agile.

Agile Transition

Transition Plan

Through the end of 2019 and the first quarter of 2020, the management team met several times with the leadership team to plan the transition to agile. In March of 2020, an all-hands staff meeting was held, and the announcement of the new scrum team makeups was provided. Personnel had been briefed that these changes were coming, so they weren’t completely caught off guard, but there was a lot of pushback on the urgency for the changes and that the transition was happening too fast (COVID had arrived and telework was being mandated at that same time). Everyone returned to their desks, and, over the next week, they assembled into their scrum teams. Scrum masters and product owners who were initially selected had been trained prior to the move and were in place to start exercising Scrum. As a program with 95 personnel, there were seven scrum teams assembled with additional support teams to help them (database management, technical order development, configuration management (CM), equipment, etc.). To help with coordination and requirements tracking, a Scrum-of-Scrums was put together where the scrum masters and product owners met three times a week to make sure any interdependencies between teams were being tracked and handled appropriately. They named this group the Product Owners and Scrum Masters (POSM).

Initial Team Structure

When the 309th originally took over sustainment of the A-10 OFP, the team structure was exactly what a stove-piped team looks like. The systems team would take requirements from the customer, refine them, architect them, and pass them off to the development team for implementation. The development team would take the requirements, develop the software, and then pass the software off to the test team. The test team would develop procedures for test, execute them, send issues back to the development team in a report, and then submit for release to the pilot community. The support teams would assist and participate in their areas of expertise. Each team performed their function and their function only as demonstrated in Figure 3.

Figure 3. Initial Team Structure

Pre-Agile Shakeup

Between 2014 and 2017 the team grew from about 30 to 80 people and the number of requirements continued to climb, reducing the ability for each team to stay stove-piped. Team leads saw a need to increase communication between teams to maintain consistency of the requirements as they passed through the process machine. Figure 4 illustrates how new teams began to form.

A “Candidate Team” structure was developed where a representative from all the major teams were assembled to discuss and “own” a candidate requirement and work together to develop the capability. They met often to discuss progress, issues with the design, test failure points, and risks. In a rough form, these were early precursors to agile scrum teams. The main difference here was the members of the main Systems, Development, Test, and Support teams were only “on-loan” to the candidate team and they only met once or twice a week to collaborate. All processes and procedures were developed and controlled by the main teams, not the candidate teams. Some individuals, due to their expertise in certain areas of the software, were assigned to more than one candidate team and had to balance their focus between candidates.

Figure 4. Candidate Teams

Figure 5. Architect Team

Post-Agile Shakeup

The main tenant of scrum is to take experts from different disciplines and place them into a team together to help and learn from each other. Instead of a team of system or developer subject matter experts, there is a team of rounded experience and knowledge. Figure 5 shows how the teams were distributed after the transition to agile.

This transition saw the end of the three main stove-piped teams and the creation of a set of teams made up of one or two of each specialty on each team. Scrum advocates for small, self-organizing teams that can cross-train members in the different disciplines to build a robust, high-producing team. In addition to these teams, an architect team was created to be the gate keeper for major requirements as they come in from the customer prior to being worked by a scrum team and provide help as needed to all the teams.

Afterward

First Six Months

It took about six months for teams to start executing scrum as outlined and prescribed like the A-Team and HRDS team had done. There was a lot of churn in processes, procedures, schedules, and leadership - some personnel leaving because they didn’t believe in the process and other team storming issues.

Defined systems, development, and test teams were dissolved and personnel from those teams were thrown together and told to work together and learn from each other instead of sitting in their stovepiped teams. Management left a lot of the discussion on process changes and improvements up to the scrum teams. They were allowed to review all processes and procedures that were used prior to the transition to agile and determine what changes needed to be made, and in some cases, the determination was to throw out a process all together. A new management team transitioned into A-10 OFP during this time, bringing a new set of ideas and views to the program, leading to a shift from allowing the scrum teams to make many of the process change decisions to management having more of a say in the process changes.

The First Year (2020)

Most processes and procedures that were followed under the waterfall acquisition lifecycle model had been reviewed and tailored to fit the scrum-of-scrum business model. Fewer and fewer complaints were being posed to management about agile, and more and more acceptance was gained. Suite 10 testing was wrapping up and Suite 11 was underway using agile development. There were some late breaking issues identified with the Suite 10 release that had been completed at about the transition time. It was decided to finish the code execution of Suite 10 using one or two of the new agile teams in an agile manner but finish all the paperwork and milestones using waterfall with the SPO. This path forward had two side effects: the software updates were completed in record time compared to how they were addressed under waterfall but caused a lot of headaches for the paperwork and wrap-up. All the headaches resulted in a firm conviction by most team members that agile was indeed the way to move forward.

The Second Year (2021)

The second year of agile implementation saw the completion of Suite 10 and a change to Suite 11 with a set release cadence that hadn’t been done before. The process refinement of the year before set the stage for A-10 OFP to begin releasing software on a regularly scheduled cycle instead of whenever the code was ready. Only finished epics were included in each cadence release. After a couple of release cycles, it was determined that quarterly was too fast for external stakeholders, and the cadence was slowed to a flight test release every four months. As the year progressed, the SPO began to recognize the potential and the impacts agile was having with developing and releasing updates as required rather than on a multi-year Suite buildup effort. The A-10 OFP team showed that, as pilot requirements changed over time, development could shift to those new requirements right away instead of waiting several years for the new requirement to be released in the next suite.

The Third Year (2022)

Suite 11 experienced three cadence releases in 2022 and two follow-up releases with the last release being the fielding release that would go to the fleet. Suite 11 was the first software release provided to the pilots using all agile software development. From the original requirements kickoff for Suite 11 in 2019, to the final release, the candidate/epic/requirements list changed dramatically. During the development cycle for Suite 11, Suite 12 requirements had been defined by ASRRC, but as Suite 11 progressed and agile began showing what the flexibility of changing requirements allowed, the SPO and pilots decided to revise the requirements list for both Suite 11 and 12. By the time Suite 11’s final release went to test, Suite 11 was a mashup of Suite 11 and Suite 12 requirements, and Suite 12 was altogether different. With agile being able to pivot to emerging needs and requirements, Suite 11 included features that the pilots needed that were identified a year or two after the Suite 11 ASRRC. That was game-changing. Agile acquisition management demonstrated how to release software capabilities at the speed of relevance.

Observations and Key Considerations

Continuous Integration/Continuous Deployment

Under the tenets of agile is the concept of Continuous Integration and Continuous Deployment (CI/ CD). CI/CD is defined as the ability to code a feature, put it through automated testing, integrate it into the production stream, and release (deploy) it on demand. Pilots cannot adapt to all the changes coming their way at continuous speeds. That is why the release cadence for flight test was determined to be four months and release fielding set at one year. The older software tools used by the legacy software builds cannot support this either, especially in an automated pipeline environment.

This past year, a team of developers has been working on automating A-10 OFP testing, proving that automated testing on a legacy system can be done, and are making inroads on automating the safety of flight testing required for every release. This effort will save hundreds of hours every release once complete. In the meantime, programs looking to go agile need to work with their customers to determine what CI/CD means. As A-10 demonstrated, agile software development can provide benefits using legacy development technology and program management tools. Programs moving to an agile approach should do what makes sense for the program and what can be sustained by the end users.

Agile Maturity

A-10 OFP started the agile journey knowing very little about how it was to be done. The Department of Defense (DoD) released their guidance on agile software acquisition lifecycle management in their instruction DoDI 5000.87, as shown in Figure 6, after A-10 transitioned to agile. A-10 OFP found that many of the implementation actions taken aligned with the guidance and the program required minimal changes to comply. Any program wanting to convert to agile development lifecycles must adhere to the agile guidelines for the method they adopt. They must execute the protocols, participate in the prescribed meetings, and perform their assigned roles as designed. Agile practices must be followed as prescribed for them to become part of the business model and for it to work. Improvement deviations must adhere to agile principles. The transformation to agile must be given a chance to succeed and should not be used to solve quick-fix issues then abandoned.

Figure 6. Software Acquisition Pathway Phase Illustration Source: DoDI 5000.87[4]

Accepting Agile

The management team spent the better part of a year (2019) discussing the merits of agile with the SPO program managers and their leadership. Only after demonstrating the results of the A-Team test run did SPO leadership see value, understand how it would work, and gave the green light to move forward.

Many personnel in the A-10 OFP development team were skeptical and not on board with the idea of transitioning to agile. During 2019, management sent several personnel to scrum master and product owner training to have them ready to pick up scrum teams when the transition was made.

Not all programs within the A-10 SPO that interface with the OFP development effort are onboard with agile, even four years after starting the conversation. Not all support agencies, like Developmental Test (DT) and Operational Test (OT), can support continuous integration and releasing software every two weeks. Agile programs must work with all the supporting agencies and customers to find a release cadence that works for the program. Not all support agencies need to move to agile, but they must be able to continue to support programs that are.

Customer Feedback

For software development to move forward as quickly and efficiently as agile allows, customer feedback is paramount. An end-user representative must be the one who provides that feedback. Programs should build this relationship with the end user, so this feedback loop is both timely and effective. If possible, integrating an end user within the development team is the most efficient.

Commit to the Process

Programs must commit to the transition. All too often, people are afraid of change and allow this fear to stop them from trying new things. Air Force Chief of Staff, Gen. Brown released the “Accelerate or Lose” [5] vision with the intent to showcase that doing business as usual is not going to help the Air Force meet future warfighter capability or air dominance. Agile business practices provide the flexibility to shift focus and priorities to meet emerging needs vs. meeting a cost/schedule model. Programs cannot allow fear of failure or change to hinder progress. Agile is all about failing early and often so improvements can be made to get the warfighters the tools they need today, not years from now

Conclusion

The transition to agile must be real. Changing process names and labeling them agile while not actually changing processes to be agile does not count. Labeling everything using agile buzz words, but not practicing the agile tenets will cause programs and customers to be discouraged, believe agile doesn’t work, and return to the old way of doing things even though they didn’t change anything to begin with. This will hinder any progress toward the goal of releasing capability at the speed of relevance.

Transitioning to agile from waterfall acquisition lifecycle management is a big leap. It may ultimately be determined that agile may not be the business model a program should adopt. The A-10 OFP development team has demonstrated to the A-10 customer base that agile software acquisition is the way to go and that a legacy platform that is 45 years old can use modern development techniques to continue to sustain and modernize the Air Force.

References

[1] DAU, “Major Capabilities Acquisition (Pre-Tailoring),” October 21, 2022, https://www.dau.edu/tools/ Documents/Lifecycle%20Chart%20-%20Major%20Capability%20Lane_print.pdf

[2] Department of Defense, “DoD Enterprise DevSecOps Reference Design,” August 2019

[3] Sutherland, Jeff (2014). Scrum: The art of doing twice the work in half the time. Crown Publishing New York

[4] DoD Instruction 5000.87, “Operation of the Software Acquisition Pathway,” October 2, 2020

[5] Brown, General Charles Q, “Accelerate Change or Lose,” August 2020, Office of the Air Force Chief of Staff, https://www.af.mil/Portals/1/documents/csaf/CSAF_22/CSAF_22_Strategic_Approach_ Accelerate_Change_or_Lose_31_Aug_2020.pdf

About the Author

Michael Engh is an Electronics Engineering Supervisor for the 520th Software Engineering Squadron at Hill AFB located in Ogden, UT. He has 11 years experience in software/hardware engineering, leading software teams as a deputy program director on various projects, and leading the agile transformation for the A-10 OFP development program. Mr. Engh received a B.S in Electrical and Computer Engineering from Utah State University and an MBA from Webster University.

Micheal Engh
Chief A-10 Operation Flight Program Supervisor
520 SWES/MXDPE
michael.engh@us.af.mil
801-586-5805

---