October is Cybersecurity Awareness Month. This year's theme, "Do Your Part. Be Cyber Smart," is a call to action for not only average users to take their cybersecurity seriously, but for enterprises to be forward thinking and address potential risks as technologies continue to evolve. This is particularly true in today's mainframe environment where legacy technology meets newer systems like containers.

Ian Coldwater, a leading expert on containers and container security, and Chad Rikansrud, a leading expert on mainframes and mainframe security, revealed they were the first to accomplish a mainframe container breakout — when a malicious or legitimate user is able to escape the container isolation and access resources on the host machine, such as file systems — at DEF CON 29 this year. In a conversation, they shared their motivations, the collaboration it took to achieve this feat, and what security lessons enterprises should take away from their efforts.

Uncovering Vulnerability

Coldwater and Rikansrud said they were motivated to complete the container breakout because they wanted to uncover the vulnerabilities in the systems and software, which they would report to the software/hardware maintainers to ultimately protect end users and improve security. The aim was to ensure the flaws were fixed before others could mount successful attacks. They also noted, "The thrill of finding a misconfiguration or software vulnerability that leads to the ability to gain access to a system or escalate privileges is exciting!"

Both have different skill sets and backgrounds, but it is clear that this undertaking would require collaboration as z/OS Container Extensions (zCX) merge the worlds of mainframe and containers. Coldwater and Rikansrud said, "zCX is an interesting and complex product, melding two different technologies built upon very different assumptions. At its core, zCX is a software hypervisor that runs atop z/OS. It runs a base Ubuntu Linux image with Docker installed. End users are provided access to the appliance via a container built and deployed by default from IBM's secure shell command line interface (SSH CLI) container, which communicates to the Linux host via the Docker daemon."

They explained, "In theory, this container is meant to be the only interface the end user has to the appliance, giving them the ability to interact with new Docker containers but attempting to restrict their ability to gain root access to or modify zCX itself."

Using their unique skills, Coldwater attacked and probed at Docker and the base Ubuntu Linux installation down into the mainframe, while Rikansrud worked from the mainframe side through z/OS, Unix System Services, and the web interface responsible for provisioning zCX up into the containers. This two-pronged attack also required some crossover work, with both utilizing their knowledge of Linux and its low-level features.

Running Docker containers directly on z/OS, they said, is a complex undertaking and that their "expectations were not terribly high regarding its secure implementation out of the box." Coldwater and Rikansrud explained, "zCX would be difficult to secure properly, especially in the initial release." They added, "When we started out, we quickly found several vulnerabilities, including some that stemmed from incorrect assumptions on how to secure Docker, outdated versions of software, and permissions on z/OS that were too relaxed."

Unmasked Vulnerabilities and Lessons Learned

Coldwater found several vulnerabilities in the IBM-supplied container setup. For example, the initial user in the SSH CLI container was a member of the Docker group, which can lead to escaping the container and gaining root access to the host in certain configurations. zCX had one of these configurations in place. Coldwater explained, "The SSH CLI container communicated directly with an exposed Docker daemon on the Linux host, allowing users in the Docker group (such as the one in the CLI container) to create and manipulate containers running on the host."

IBM had tried to restrict access to host resources through the Docker authorization plugin called zcxauthplugin, but that plugin neglected to protect the Docker Engine API, an HTTP-based API that comes with Docker and can be used to issue a number of commands through the Docker command line tool. Coldwater used Engine API to bypass the zcxauthplugin and send commands to create a container that allowed access to host resources, thereby escaping the container and gaining access to the host from there.

Rikansrud, meanwhile, discovered that, by default, the Unix System Services files used to provision and build the zCX instance lacked basic file-system security. "This would allow any malicious user to backdoor these files such that any container created from them would be compromised from the start," he said.

"Enterprises implementing zCX," Coldwater and Rikansrud said, "should be familiar with more than how to secure the z/OS address space, users, USS files and directories, and zOSMF configurations." Moreover, they added that firms "also should know how to properly secure a Linux instance and Docker itself. The implementation is very complex and cutting corners anywhere could allow for an insecure implementation."

Be Cyber Smart

Enterprises should actively educate themselves on what security in their systems should look like and how to properly secure or verify that their systems are locked down in accordance with their own cybersecurity standards. Further, Coldwater and Rikansrud shared, "It is incumbent on the maintainer and purveyors of software and hardware to make sure end users know what security on these systems should look like. Buying a secure, or black-box style, product does not relieve enterprises from their duty to ensure assets are protected."

Enterprises should be particularly cognizant of places where differing assumptions and practices could lead to things falling through the cracks. "One example is patching. In open source, software tends to be deployed more often and patched more frequently, whereas 'move fast' has not traditionally been a tenet of the mainframe world, which deploys and patches less often," Rikansrud pointed out. "This can lead to a situation where there are known vulnerabilities in mainframe systems that have not been patched on other systems for a while."

With that aside, Coldwater and Rikansrud said, "implementing containers on mainframe could have tremendous business value and provide a load of utility for the business, but care must be taken that the right skill sets are present for both the z/OS side and the Docker/container side of the equation, as neglecting either could spell disaster." Containers on the mainframe mix old and new technology, and they warned that "unexpected combinations can have unexpected results," which is why security should continue to be a top priority.

Find the full session from DEF CON 29 on YouTube, or check out our other stories on cybersecurity.