Have you ever seen a dump taken on your system and wondered whether you needed a plumber? We’ll share with you some tricks of the trade to help you become an apprentice plumber for z/OS dumps. Interactive Problem Control System (IPCS) can be utilized to plunge ahead and provide insight into what contributed to the clog, including a history and timeline of events. Soon, you'll be well on your way to flushing out bugs!
Get the Toolbox Ready
When faced with a potential catastrophe, don’t run away! Put on your galoshes and jump into the dump! First, identify the dump to IPCS by specifying the dump dataset name in IPCS Option 0 (Defaults panel). Throughout the article, we’ll highlight some of the tools needed to get the job done.
Did We SLIP and Fall or Recover Our Balance?
Now that IPCS is set up to view the dump, your first order of business is to determine what type of dump you’re dealing with. Two types of dumps you’ll likely encounter are:
- Recovery dumps
- SLIP dumps
When an ABEND (ABnormal END) occurs, the z/OS Recovery Termination Manager (RTM) gives control to established recovery routines, which collect the necessary documentation (e.g., dump) for debug and try to recover from the ABEND. We refer to “recovery dump” as a dump taken by a recovery routine. On the other hand, SLIP (Serviceability Level Indication Processing) is a very powerful diagnostic utility that lets you trap on an ABEND or message to collect a dump.
Now, it’s time to get your feet wet with this dump! You can issue the “IP STATUS” command from the IPCS command line to ascertain what type of dump you have. Since this is the first IPCS command you are issuing against the dump, it will cause the dump to fully initialize. Once initialization is complete, the “Dump Title” at the top of the command output will allow you to determine the type of dump. Recovery dumps typically include the ABEND code in the title, and SLIP dumps include the text “SLIP DUMP” in the title along with the ID of the SLIP trap.
Investigate the Clog
Let’s continue gathering our relevant diagnostic data. Did an ABEND cause this dump?
There are two categories of ABENDs: System ABENDs (three-digit hex code) and User ABENDs (four-digit decimal code). System ABENDs are the result of errors detected by the operating system, and user ABENDs are the result of errors detected by an application. For a recovery dump, the title will often distinguish the ABEND type. For a SLIP dump, the “IP LIST SLIPTRAP” output will differentiate the ABEND type (C= parm). If the SLIPTRAP output instead shows MSGID=, your dump was collected in response to a message.
Let’s Dive Deeper!
Start asking questions and looking for answers in product documentation. Review the z/OS MVS System Codes manual to investigate the meaning of a system ABEND code and its associated reason code. Consult application-specific documentation to gain a deeper understanding of a specific user ABEND code. Use the LOOK@ tool to find the meaning of an IBM message. Product documentation will often provide tailored operator or system programmer responses. Can you take action to save the day?
Who Is Responsible for the Clog?
At this juncture, based on the information you’ve obtained about the ABEND or message that produced the dump, you need to determine whether you have enough information to resolve your own issue. Is this a DIY (Do It Yourself) job where you can remove the ball of hair clogging the sink and be a hero? Or is it unclear from the ABEND or message what is wrong and how to resolve the issue? In the latter case, it is time to call the plumber!
But which plumber should you call? Does the ABEND or message point to a problem with an in-house application, IBM code, or OEM vendor code? The “IP STATUS REGS” command provides you with the PSW (Program Status Word) that points to the code that was executing at the time of the failure. For a recovery dump, the PSW points to the code where the failure occurred, and for a SLIP dump, it points to the code where the SLIP matched.
The “IP WHERE” command can help you map the PSW address to an offset within a CSECT (code block) or a load module (collection of CSECTs). If the PSW address points within the Nucleus in global storage, then “WHERE” will provide you with the failing CSECT name plus an offset. For all other storage areas, “WHERE” will only provide the failing load module plus an offset. In the latter case, you will need to browse storage in the dump (IPCS Option 1 – “BROWSE” panel), backing up from where the PSW points to look for a module CSECT eyecatcher, which typically includes the module name, maintenance level/compile date, and copyright information. Alternatively, you can run an AMBLIST report to map an LPA or private load module plus an offset to a specific CSECT name plus an offset.
Before reaching out to IBM for assistance, you should search IBM Support for known issues and technical documentation. Be sure to construct your search with standardized keywords for all pieces of information you have gathered from the dump including the ABEND code (ABENDxxx, ABENDUdddd), message (MSGIDxxxx), and CSECT name. INFO APAR II07428 contains a list of common keywords.
Common Plumbing Mishaps
You may encounter unexpected issues when attempting to browse virtual storage in a dump. A relevant address space or other important area of virtual storage might be missing. Perhaps that storage was not requested for inclusion into the dump by a recovery routine or SLIP, or the dump was partial and did not successfully capture or write all requested virtual storage. Always review your SYSLOG output for either an IEA611I or IEA911E message to confirm whether your dump is complete. Lastly, ensure that all intermediate copy and FTP steps of the dump transfer are successful.
What Led to the Clog? Were There Prior Clogs?
For many types of problems, the history of ABENDs and messages leading up to the dump is very beneficial and can provide additional clues. You can issue “IP VERBX MTRACE” to view a snapshot of SYSLOG data just prior to the dump, which can be useful to see if a job was started, if a message was issued, or if a command was issued just prior to the problem. The “IP VERBX LOGDATA” command will provide you with a history of ABENDs (LOGREC data) leading up to the problem. Were there other ABENDs immediately preceding this dump that could possibly be related?
When investigating clogs (e.g., dumps), always have your IPCS plumbing toolbox handy! The tools provided by IPCS can help you identify the key pieces of evidence required to either unclog your own drain, or to instead reach out to the appropriate plumbing professional.
If you want to learn more about handling system dumps, check out the SHARE Proceedings. Not a member yet? Join today!
John C. Shebey III is a Senior Technical Staff Member (STSM) Support SME at IBM Poughkeepsie. He has been in z/OS Support for the last 27 years and is the IBM z/OS Software Service representative for SHARE. He holds a BS degree in Mathematics from Virginia Tech.
Laura M. Sperling is a Senior Support SME at IBM Poughkeepsie. She has been in z/OS Support for the last 22 years and is a frequent presenter at SHARE. She holds a BS degree in Computer Science from Binghamton University.