Mainframe testing continues to be essential to the continued success of the platform. Petr Vacula, mainframe product manager at Broadcom, and Josh Dorfman, a UX research strategist at Broadcom, explored manual and automated testing in their session, "Breaking the Habit of Manual Testing," during the SHARE Virtual Experience 2021 (August 9–13). The session discussed the techniques and tools that are used for mainframe testing historically and today, why manual testing remains prevalent on the mainframe, and how automation can speed up testing and provide additional benefits to organizations.

Vacula says, "DevOps maturity differs between shops and organizations with more mature processes have higher test automation." However, he says that generally about only 10% of all mainframe testing is being automated in some fashion. "For one automated test case, there are nine that someone needs to drive manually," Vacula explains. "The type of applications not covered by test automation vary through the whole application stack from batch reports, through online transactions, to backend ISPF panels."

Vacula points out that the manual work setting the environment up for intertwined systems takes the most time, while actual testing takes a fraction of the overall time. "This inevitably drains your very skilled SMEs precious time," he says. "While you might get a good return on investment from automating testing of such systems, it might not be a good choice if you're just starting with test automation." He recommends that organizations looking to automate their testing should build up their skills. They need to determine which application they know best and where they can get the biggest time savings, compared to the effort of automating it.

Biggest Challenges of Manual Testing

Surprisingly, the biggest challenges when moving an organization away from manual testing is not technical. "Usually it’s people who hold back on that," says Vacula. "It’s resistance to change, perception, cultural silos, and skills gap." He adds that once you understand these challenges and keep them in mind, moving to automated testing can become a smoother transition. "Only then you might really start thinking about the technical how aspect," advises Vacula.

The first step is for an organization to think about the benefit of automating a particular application's manual tests. Vacula says, "It might be easy to automate the testing for a given application, but if it will not yield any real benefit, what's the point of the automation?" He says that applications that are stable and don't have any new additions coming in are not good candidates for automated testing. "Starting with estimates on what might be the savings will give you an idea of whether it will be worth the effort to automate the application's testing," explains Vacula. "Usually it’s easier to automate applications that have some callable interface on their own. This way you don't need to figure out how to drive the application's specifics. For example, driving a batch application submission is as straightforward as just submitting a job."

The Move Away from Manual Testing

Vacula advises that organizations first think of the potential barriers to automated testing. One example he cites is if there is a clear separation of duties when it comes to development and quality assurance. If so, he says that an organization may have some cultural silos to break down first, and then the firm will need to determine if they have the skills necessary to transition to automation. If additional skills are needed, the organization will need to invest in their teams.

Secondly, organizations will want to avoid reinventing the wheel, explains Vacula. "Where possible, leverage in-house expertise with automated testing practices and tools," he says. "Connect with your enterprise DevOps champions or center of excellence."

Third, Vacula says to treat the mainframe like any other platform in the hybrid ecosystem and use existing testing frameworks. He adds that organizations also should consider leveraging mainframe APIs. Finally, organizations will want to turn to teams that are already primed for an automation mindset, says Vacula.

Automated testing saves time in the long run, he says. Vacula explains that eliminating manual tasks frees team members to focus on more higher-value, business-aligned activities, including business innovation and application modernization (e.g. containerization). "The opportunity costs of manual testing is significant," he adds. Additionally, automated testing can surface defects earlier in the process, which can reduce the overall costs by up to 640 times. Vacula also says that effective test automation elevates the quality standard improving customer satisfaction and confidence in your team.