Broadcom's experiences with an apprenticeship program.
Becoming a professional mainframe software engineer is hard
Becoming an experienced, professional software developer on today's modern mainframe presents a unique set of challenges. At Broadcom, we certainly strive to use modern technology, but much of our mainframe product software has evolved over decades and its core is written in very low-level languages by today’s standards, including High Level Assembler, Metal C, and C. Being a member of our mainframe engineering team means not only programming in these languages, but understanding the vast amounts of legacy code written in them. Language aside, programming on IBM’s z/OS operating system presents a fairly unique architecture when compared with today’s cloud stacks. Mainframe engineers must be well versed in this vast architecture and the security and integrity considerations it imposes.
Not too long ago, I had been involved in analyzing the root cause of bugs that were injected into some of our mature software products by newer engineers. I was surprised to see that most of the bugs did not result from coding errors, per se, but from lack of knowledge in some of the idiosyncrasies of the underlying system. When I asked some of our experienced mainframe gurus how these engineers could have known to avoid these problems, the consensus was interesting. These gurus did not believe there was a course or some book the engineers needed to read, but felt the best way to avoid these problems would have been to talk to someone more experienced in the platform. These experienced engineers all spotted the issues right away based on their years of practical experience programming on z/OS.
This got me thinking. We needed a way to get this next generation working more with these experienced engineers to begin transferring some of this “tribal” platform knowledge. These stories reminded me of many of the traditional trades where an apprenticeship was used to train new master craftsmen. In a trade apprenticeship, it is not enough to read about the trade or even watch a master at work. In a true apprenticeship, the master and the mentee must work together actually doing the job. This line of thinking is what eventually led to the creation of an apprenticeship program within Broadcom’s mainframe team.
Mainframe Apprenticeship Program
At Broadcom, we are investing in the mainframe for the future. We are fortunate to be growing our mainframe engineering team, and that our leadership — at all levels — recognizes the importance of mainframe training and education. As such, there are a number of excellent programs available to all employees, including our Associate Software Engineering (ASE) training program that all new hires must go through. There are lots of other education options as well, from internal courses, to commercial online courses, and even regular lunch-and-learn sessions presented division-wide by members of our team. All of these education options have their place, but we discovered that we really needed to put something in place that encouraged in-context learning alongside an experienced mainframe expert. This was the motivation for creating the apprenticeship program.
Our apprenticeship program is pretty straightforward, with the goal being to pair mainframe experts with early career engineers who are motivated to become the experts of the future. Here’s how we implemented it:
- Pick the pairs. This seems pretty obvious, but is perhaps the most important factor for success. In choosing these pairs, it was critical that the mentees have a desire, perhaps even a passion, for learning the details of the mainframe. This is largely a self-directed program and this motivation is critical to keep things going.
Likewise, you need mentors who are patient, good teachers, and have years of experience on the platform. What we have found is that even senior engineers who are perceived as a bit “crusty” can often turn out to be great teachers and mentors when given the opportunity, and when paired with someone truly interested in learning from their experience and expertise.
In our organization, we identified these pairs in conjunction with our development directors and managers. We involved the development directors every step of the way in this program. They helped find the right personality matches and also ensured that development management was invested in making this program work. If possible, we tried to select co-located pairs, but this was not always feasible because of the distributed nature of our teams across sites, and because of the current pandemic situation.
- Pair ‘em up and kick it off. After we identified the pairs of participants, their managers contacted the engineers, validated their interest in participating in the apprenticeship, and asked them to set up time to meet with each other at least for one hour each week, initially. In these initial meetings, the goal is to allow the pairs to get to know each other and start to build a relationship. These meetings were private, informal meetings between each pair. The only thing we asked at this stage is that they meet, and that they began to think about assignments where they could work together in the context of “deep” mainframe.
After giving the pairs time to get to know each other, we scheduled a kickoff meeting with all the pairs. I ran this meeting as CTO of the division, but the meeting could be run by any senior leader. The main goal of the meeting, in addition to establishing the goals and guidelines of the program, was to make sure that each participant understood how important these apprenticeships were for our division’s future health. Above all, we wanted to let them know they had complete support from their senior leadership.
- Enable collaboration. Ideally, this would have been face-to-face, but those are not the times we live in right now. To make up for this, we created chat rooms and encouraged teams to share their experiences. The chat rooms, at present, are not used much if I am being honest. I am thinking this could be improved by posting some questions every once and a while to stimulate discussion. To get deeper collaboration across the teams, we needed to schedule checkpoint meetings … more on these later.
- Pick and schedule “in-context” assignments. This is also an important aspect of the success of this program. Teams were tasked with coming up with assignments related to mainframe to work on together. Coming up with these assignments was not always easy, since some of the participants were not in the same product areas and some of the mentees were not even assigned to deep mainframe-related projects. The teams got creative in coming up with ways to work together. Some teams picked stretch assignments from their product backlogs. For example, one team is working on modifying CA-GEN to generate restful web services that interface with mainframe-generated code. Another team is working on an internal tool to scan code for z/OS integrity and security violations. And, yet another team is exploring federated security tokens across mainframe and cloud.
In addition to these contextual projects, teams have started scheduling times to present to each other on mainframe topics of interest. In our Prague lab, several teams get together every Friday afternoon to explore a mainframe topic in depth. Pairs have also started sharing presentations and documentation that they have found useful.
- Check in and adjust occasionally. We have also found it useful to get all the apprenticeship teams together about once every two months in an hour-long virtual check-up meeting. In this meeting, each team talks about the status of their projects and shares what is working — or not working — for them with the other participants. These meetings are necessary. They hold the teams accountable and provide extra motivation for them to make progress. In addition, they are a great way to share stories of success (and failure/discovery).
We have been fortunate that most of our pairings have worked well. But this may not always be the case. Teams may not progress for a variety of reasons including personality clashes, lack of motivation, personal reasons, or changing business needs. In one instance that did not work for us, the apprentice did not have the interest we had predicted in learning deep mainframe and was not progressing. We ended up concluding this relationship and adding another apprentice to the program. It's important to remain agile, and make adjustments as needed to keep the program healthy.
Results so far
It's still early for us with this apprenticeship program, and we are in it for the long haul. The program has been running for about eight months, and we are beginning to see some successes. The projects that pairs are working on together are starting to progress. We’re seeing exactly the type of in-context learning that seemed so elusive at the outset. As teams learn, mentees are getting even more motivated about their careers in mainframe.
Feedback from participants in the program has been very positive. In general, they appreciate the fact that leadership is committed to allocating the time and effort to let them develop these apprentice relationships. One team has even presented a session at SHARE Virtual Summit on their positive experiences in the program. We have also had requests from engineers not in the program to join, and we are in the process of expanding the initial pilot to more apprenticeship pairs.
At Broadcom, we plan to continue these apprenticeships well into the future based on these early results. The program is igniting a spark for working on the mainframe with our newer engineers and they are gaining invaluable experience working alongside the experts that helped build the mainframe platform. In the process they are making lasting relationships across diverse backgrounds and generations. Perhaps a similar program might yield results within your organization?