Agility is a key driver of business growth in today's rapidly evolving markets. DevOps helps enterprises achieve that agility by shortening development life cycles for new applications and ensuring businesses have the latest technology leveraging their data. DevOps is more than a set of practices that combines software development and IT operations. It also is a shared culture of workflow automation, shared ownership, and rapid feedback, all with the aim of advancing enterprise technology. After SHARE Dallas, several DevOps experts offered their thoughts on DevOps culture, mainframe modernization, and the future of the discipline.

IBM Fellow Rosalind Radcliffe, DevSecOps chief technology officer at IBM, said that from her experience, working in systems management before the term DevOps even existed demonstrated a need to improve client experiences, particularly when implementing solutions. "When I moved into the application development space it was obvious there was a need to bring those organizations together," she said. "DevOps is seen as a way to empower various teams to improve their processes, but as this has grown the complexity of the solutions within the organizations has grown and the duplication of effort across teams has increased."

Vaughn Marshall, senior principal product manager at Broadcom, said, "Fundamentally, I have always hated rote, repetitive work. As a developer, whenever I encountered it, I've tried to make it something I could script, automate, or otherwise minimize. Over the years, consulting, learning how different organizations do things, and watching how the industry has evolved, I've found DevOps is a natural fit for this way of thinking."

Venkat Balabhadrapatruni, distinguished engineer of enterprise DevOps at Broadcom, said, "The thing that attracted me to DevOps is the fact that it's about people and culture, and successful DevOps adoption requires driving that cultural change, which is the most challenging aspect. I relish that challenge." Because z/OS customers need new applications to do business, Jack Opgenorth, senior technical consultant at IntelliMagic, explained, "My interest in DevOps is to ensure application development processes contain governance that includes good stewardship of infrastructure resources."

DevOps Transformation

At Broadcom, Vaughn said DevOps tools have helped automate and drive its mainframe development process. Through these tools, "We've embraced Agile and moved to an automated, continuous delivery process for new product features," he explained. Because the process continues to evolve and employ new tools, Broadcom has "added new layers of security scanning, driven via integrating our mainframe pipeline to off-host CI/CD (the combined practices of continuous integration and continuous delivery or continuous deployment - see https://en.wikipedia.org/wiki/CI/CD) tools facilitated by the open source Zowe CLI project," said Vaughn. "In light of the recent supply-chain attacks in the news, it was critical we ensured we applied the latest security practices to our pipelines."

Venkat pointed out that Broadcom has several teams that have embraced DevOps, including those working with traditional products and new "green field" products. SYSVIEW, for example, was a traditional monitoring solution that had little-to-no build and test automation. "Once the team of new hires and experienced developers fully embraced DevOps and open source, SYSVIEW now has a fully automated DevOps pipeline that uses open source tools," he explained. "The automation of builds and tests helped the team to develop and deliver quality code faster, but also freed up the team's capacity to focus on delivering more value to our customers. The SYSVIEW team's transformation is a perfect example of how a traditional product written mostly in high-level Assembler (HLASM) language can embrace DevOps and open source to transform software development and delivery lifecycle."

In terms of z/OS, Opgenorth explained that mainframe telemetry data can be integrated into the DevOps workflow. Change detection, for instance, allows developers to identify and resolve application performance issues by monitoring statistically significant changes in an application and system behavior. This also enables businesses to spend less time on the phone determining where problems could be (see figure below). He added, "By including performance data into the feedback loop, we were able to prevent production performance degradation."

Figure: Address Space Change Detection (IntelliMagic)

Additionally, Radcliffe pointed out that IBM's DevOps transformation teams have now moved from yearly or longer releases to a continuous delivery model, which enables new functions to be delivered to clients quarterly for many products. "This change allows clients to adopt capabilities more rapidly if they are ready, and it removes more of the manual tasks and improves the automation allowing teams to focus on building more business value."

Tips for Tackling DevOps Transformations

There are several considerations that teams need to take account of before tackling DevOps transformations, according to Radcliffe. "There's no reason every development team should be building the same capabilities," she said. "For example, the creation of the Software Bill of Materials (SBOM) will be required by legislation soon, if not already. However, having every team create a set of capabilities to provide the SBOM is not an efficient way to deliver business value." A centralized approach for the developer experience makes the most sense. She added "This is an opportunity for inner-source model for development capabilities."

A second criterion Radcliffe outlines is the development of capabilities as individual parts to help as tools change. "One thing that is clear in the industry is that tools are changing faster than ever, and you want to be able to pivot to new tools as required," she explained, "but if you have built capabilities as tasks that can be shared, then substituting one pipeline orchestrator for another is no big effort."

Before undergoing a transformation, Opgenorth said the project will need a key stakeholder champion within the enterprise and early participation by stakeholders responsible for availability and security of your applications to ensure they are included in DevOps improvements. He also advised that "enterprises establish governance within DevOps that ensures the business benefits from the investment. Stakeholders also should be empowered to share successful techniques across business units by standardizing certain operations pipelines."

Vaughn cautioned that there is no one-size-fits-all approach to DevOps transformation within a single organization. "Things vary from team to team, app to app, and the best approach is the most flexible one," he explained. Teams should leverage the platform tools they have, but Vaughn added that some additional components may be necessary to bridge the gap between the mainframe and DevOps tools. For the most part, however, the mainframe does work well with modern DevOps tools the enterprise already invests in.

"Other key things to keep in mind," he said, "is that teams can layer the new onto the old to keep moving forward without 'reinventing the wheel.'" Transformation also goes smoother when there are internal experts who offer insight, and that advice is bolstered by formal and informal DevOps Centers of Excellence (CoE) that work with and onboard teams. "If there is already a DevOps CoE for distributed tools, see if mainframe can be made a participant," said Vaughn.

Venkat agreed that a "DevOps CoE can help scale the DevOps transformation across various teams within the organization and act as a central team who can assist teams [to] start and ramp up on their DevOps journey." He said, "DevOps transformation is about people and culture, and the journey towards breaking the silos. Tools and processes are enablers, but driving cultural transformation with a mindset of continuous improvement and continuous learning is the most important aspect. Engaging with your teams, understanding their preferences, and coming up with a transformation plan with minimal disruptions is key."

Venkat offered the following tips for DevOps transformation success:

1. Understand the business need
2. Identify where the bottlenecks are to achieving business goals
3.  Address those bottlenecks with a well-executed plan that includes precise measurements for success
4. Adopt metrics that are meaningful to the organization
5. Pivot as needed in order to continue making progress toward the goal

"The goal of DevOps transformation is to deliver quality software at the speed dictated by the business needs," he said. It also ensures repeatable tasks are automated as necessary and helps reduce waste. "The key for successful automation is to start slow and scale based on the learnings," Venkat added. Moreover, he pointed out that "opening up the infrastructure with a set of APIs can enable standardization across the enterprise and eliminate the need for platform specific tools and processes."

Opgenorth advised, "Start small enough to win, and choose something meaningful to your champion. Be prepared for push back and recognize that good processes should make development easier, which drives adoption over time." Radcliffe agreed, noting, "In any transformation you need to take the time to slow down to speed up."

Agility Requires Change

Interviewees agreed that DevOps transformation is a journey, and choosing manageable tasks at the start can set your teams up for success. "Instead of boiling the ocean, look for areas where small changes can get quick wins with novel processes, and then iterate," added Vaughn. "For instance, adding static application analysis or source scanning to the mix by layering on top of existing build automation is a fairly straightforward first step and can help get buy-in from the affected teams because it gives a clear return on investment. If the teams see value and you work to minimize disruption, things always go smoother."

"You also need to understand the current waste in the system, the manual tasks, and the things that get in people’s way and frustrate the efforts to deliver value," recommended Radcliffe. "Start there with continuous improvement. People will change with the right support from management and when they see the value to them in their daily work."

Make each team an ally in the transformation process by listening to their concerns and communicating how the process will achieve their team’s goals. Vaughn suggested executive level support as a way to help move things along, but DevOps teams should never forget to demonstrate the value the new processes will provide each team and the overall enterprise. "Forcing change rarely gives a favorable outcome and may even be responsible for sending some of your best developers out the door," he warned.

Venkat pointed out that "DevOps is a perfect catalyst that enables mainframe modernization and allows enterprises to use the standard set of tooling to develop, build, test, and deliver quality, multi-tiered and multi-platform applications to match their business needs and speed."

Radcliffe added that tools provide standardization and automation. But it is key when choosing tools to "pick the standard for the organization, not the one tool for distributed, the one tool for mainframe, or the one tool for development or operations." She continued, "There are some differences, but only include variation in the tool stack when absolutely necessary. It also is important to simplify the tool stack when possible. For example, using a single source code manager provides one point for control, and a single pipeline tool allows for a single way to integrate approvals and capture data."

According to Radcliffe, "Forward-looking companies, including IBM, are now looking to create a centralized product team to support various teams. This product team can then implement, only once, strategic capabilities for use by the product teams, but it will still allow for flexibility for each team. This will help as we all move forward with DevSecOps to ensure we are creating a secure software supply chain." She added, "If we all focus on this continuous learning and improvement, we can learn from each other and deliver greater business value to all our clients, while having more fun. It’s the founding principle of SHARE, 'it's not an acronym it's what we do', we SHARE what we have learned. This practice of coming together and learning from each other is what makes the industry move forward." DevOps has a bright future as enterprises move to the hybrid cloud environment and modernize their mainframes to meet new business needs. DevOps is the way that enterprises can achieve the automation and agility they need to serve their customers more effectively.