“I don't like the term DevOps sort of, because it's really not dev and ops, it's really business and development and Q&A and operations. It's everybody that has to get this transformation.”


Ready Player OneSpace InvadersAS/400MVS XAISPFPLX ProgrammingIBM Z

DevOps for Enterprise Systems

The Mainframe DevOps Team Saves the Day

Test Automation for Mainframes

Companies Mentioned

IBMUniversity of FloridaTivoliDevOps Enterprise SummitGoogle

Rosalind Radcliff, Distinguished Engineer and Chief Architect for DevOps on z Sytems at IBM, works with DevOps on mainframes.  In this Innovator's Journey, I talk with Rosalind about her background in technology, what she has worked on at IBM and what she hopes to accomplishes within the next year. This conversation was recorded at the 2016 DevOps Enterprise Summit in San Francisco. 

Mark: Did you do technology as a child?

Rosalind: I'm old enough that there wasn't a lot of technology when I was growing up, so in my middle school we got access to computers, sort of. It was really a mainframe in the backend, but we didn't know it was. It was this thing that we could use this teletype for and get access to.

Mark: When was this?

Rosalind: ‘79, maybe. '80? Somewhere in there. In my high school, we still had access to that as well as the early Apples. We had some really early computers and did some playing. We did Space Invaders, and we did those kinds of things.

Mark: Have you read Ready Player One? Have you seen that? You want to put that on your list. He writes a novel that incorporates all of the games and TV shows and music from the 1980's. Everything is mentioned. Yeah, it's very cool. Very cool.

Rosalind: Yeah, I did computers when I was young. I got to play with them because I enjoyed them and they were easy.

Mark: At school? Where did this thing show up?

Rosalind: At school. I was in a superior ability program -- the kids over to the side they didn't know what to do with because we got bored too easily, so we could go do extra things. I did the math track as a kid, so we would go take math tests, just for the fun of it .Although it was a little strange, it was fun and it was easy, so when I went to college, I did computer science because I thought that might be the easiest degree. It seemed a little strange to people.

Mark: Where was that?

Rosalind: University of Florida. One of the top 13 computer science schools at the time. 1987 is when I graduated.

Mark: So the Net hadn't really started yet. It wasn't on the radar at all.

Rosalind: No. This is long before you generally had computers at home and my first computer had two floppy drives. I don't want to even think about how much memory it had, because that would just be so depressing.

Mark: I remember, I had one of those, too. Yes.

Rosalind: But it was a toy. It was fun and I could play and it was easy, so that's why I did it.

Mark: When I first started doing it, I didn't know the power that was behind it, what it was capable of. How soon did you figure out that this thing is pretty powerful?

Rosalind: I don't know that I really had a clue even when I started with IBM in some ways. I knew it was powerful, but I hadn't totally translated how much it really could run the world, and how much it really was going to transform the way everybody thinks. I never would have imagined the iPhone. The concepts of that, it was the big system that was going to help improve business.

Mark: So AS/400, is that where you started?

Rosalind: I started on Z actually. Well, it wasn't Z then. I started in ISPF development at IBM straight out of college and it was MVS XA at the time, so a lot older.

Mark: What were you doing?

Rosalind: ISBF development. I was hired into IBM to be a developer on our menuing system and did that for a little while. I did development. I did testing. When we started in the company, the assumption was you didn't know anything, which was probably true, so we had to go learn Assembler, and we had to go learn PLX, because a lot of the IBM internal software is not actually written in a language that's externally available. I started there and got to learn how to use Z, well it wasn't Z, but how to use the mainframe and how to do things on that. I've had lots of different jobs since then. I worked on the System/38 right before it became AS/400, then I had AS/400 time as well. I have done lots of different things and lots of different experiments.

Mark: Do you remember the first time you heard about DevOps?

Rosalind: Not totally. I don't know where it was. I do remember I heard DevOps and went, "Yeah, okay, that, in some ways, is nothing new, but maybe we can make something out of it. I was in IBM for a long time in the Tivoli organization, so I did the ops side and then someone decided that I had the right skills to move into Rational on the development side. I'd spent a lot of time saying it's the developers’ problem, and then I get to go to the other side and I can't say it's ops’ problem. That just doesn't work. When that term DevOps came about, it was really, "You know, I've done ops, I've done dev. Let's break down that silo." I have been trying to break down those silos ever since. Even before I moved into Rational, because we had to fix the problem.

I actually did IBM SOA management strategy. In SOA management, we were crossing the boundaries between development and operations already, trying to get them to understand these were services we were deploying. We had to have the operational data in development to be able to understand it. To meter it, to manage it, all of those things, and so it actually kind of made sense to me and I went, "Yeah, let's break down these walls." 

That's why I do a lot with DevOps, because it just makes sense. When I think about the Z platform, which I care a lot about, I look at DevOps and go, "That's a way of getting us out of the rut that we're in." Because many people have just been doing development exactly the same way for the last 30 years, unless the Z environment changes, we have a problem because kids don't want to do COBOL. It's not that they don't want to do COBOL, it's that they don't want to use that green screen and they don't want to do waterfall development. They want to get something done fast and they want to see the value quickly. None of that can't be done on Z, it's just not the way it's always been done.

Mark: You're one of the few, if the one, that is recognized for DevOps and mainframe in the same breath. Have you drunk the DevOps Kool-Aid? 

Rosalind: I am definitely one of the people who believe that DevOps has to matter on the mainframe side. From my perspective, the idea of breaking down the silos has always been the right answer. I really do think the cultural shift needs to take place. However, I don't like the term DevOps sort of, because it's really not dev and ops, it's really business and development and Q&A and operations. It's everybody that has to get this transformation. Other than the word, absolutely. Because taking Agile and Lean and applying it across the business is the right thing for all of our platforms and all of our businesses, so absolutely. If I can get even a small percentage of all the Z customers to move, then I've accomplished something.

Mark: How large is that base for Z?

Rosalind: I have no clue other than 92 of the top 100 banks, 100 of the top 100 insurance companies. It's those kinds of numbers. I don't really know how many total customers have Z, but it almost boils down to everyone that's big, well, anyone that's big and has a need for a high transaction throughput. Most manufacturing companies, car manufacturers, and airline manufacturers. All of those have a need for high through-put and so you see Z systems. If you're a Google and you don't care about the transactions that is one thing. It's a strange way to think about it, but a bank cares. Every single transaction has to complete and has to complete exactly once. But if you think about Google, it's huge, but if the transaction fails on three boxes nobody knows.

If you're huge and you have a transaction need, then pretty much you have Z or you have IBMI. You have one of the two.

Mark: Using the term DevOps, and I have an issue with it myself, it seems to isolate security outside of that process. 

Rosalind: Security and audit seem to be left out, which is one of the things that I try and get the clients I talk to to understand they need to bring security in from the very beginning. They need to bring audit in from the very beginning. Especially in large-scale companies, where the regulators and the auditors rule in many ways.

There's so many regulations around financial transactions or insurance transactions or, take your choice, that you really need to bring auditors, security, and everybody in from the very beginning so that they're bought in on the fact that, actually, this is better. They've got a better audit trail. You can do security scanning much earlier in the cycle and find the problem earlier rather than at the very last minute when now you're causing the business to miss its deadline.

Mark: I have a difficulty figuring out why it's so hard for people to understand that security begins at the beginning, with everybody else, because everyone I talk to says the same thing. Who's not agreeing with this?

Rosalind: I don't know that anybody is, although I'm sure there is somebody not agreeing. I think in many companies, the way they think about security is separate. It's a separate group and it's a separate responsibility and so they're just not tied in.

Mark: Is it an organizational chart issue?

Rosalind: It may be entirely an organizational issue in some organizations. In some organizations, it's just they want to be different. 

I have run into many organizations in which the security team says, "We have to do things differently because we're security," and I'm like, "Why?" If you're different and you're separate, then you're slowing things down or you're not getting your data as fast. The other problem a lot of times is with companies who pay somebody to do their security scans, to do their whatever, because they must do certifications. Got it. But when they send code to them, it's going to take a day. My DevOps pipeline doesn’t work if I'm sending it off for a day, so they have to figure out ways in this pipeline to let the pipeline go but send it over there and hope that it comes back approved.

There's things like that that just must get fixed. We must figure out better ways to scan to get information faster and identify the errors or identify the possible errors. If we do it sooner, in smaller batches, I think we can help.

Mark: I have a difficulty in general with the word scan. It seems to me that if we do it properly, it is part of the process itself. You’re shaking your head yes to that. Can you talk about that a little bit? 

Rosalind: We should do enforcement of the code in the first place to say, "You shouldn't do these kinds of things." You should have code rules in the first place that identify that's just a bad practice. The problem is developers are going to write code and you've got to check it at some point in time. Even if you tell them, "That's a bad practice," you may end up needing to scan their code.

I think it should be scanned before it's delivered. I mean, let's check it as early as possible. A lot of people say that slows it down or take your choice of reaction. Somewhere, really early in the process, you need to look at the code and understand it. But there's also the process of scanning the system. You not only have to scan the code, but you have to run it and you have to see: Does it run? Does it have these vulnerabilities? Can I break into it? 

There are different things you must do from a security stand point. I think that should be all automatable, so that there is no need for a user to participate in this other than to look at the errors to fix them.

Mark: We've actually talked to people, our customers, that will set up automated rules and that way if you've got, say, a thousand instances that have to be checked, then you only have to manually check 200 of those once you automate the process.

Rosalind: We, I mean internally at IBM, we do automated security scans on source code and it's, depending on the team, it's really early in the cycle or it's later. I prefer the really early in the cycle. But you’ve got to decide which of these rules are real and which aren't. That's the other problem with security scanning. It's not a perfect science by any stretch of the imagination. There are things that could potentially be a problem, so they get flagged.

That actually frustrates development more than anything else, because now I've got these errors that really aren't errors, because they aren't errors in most cases. Sometimes they are. Unless we can clean up some of that, we end up with developers not trusting the process or not wanting to do it. We end up with this, "Security's causing my problem. They're slowing me down."

We have to transition people away from that mentality. We can do that if we just have security people on the team, if we can just break down those barriers so that people aren't still thinking it's different. If everybody gets the right mentality, we're all better off.

Mark: Using Gene Kim’s terminology, how much unplanned, unscheduled work are you having to deal with?

Rosalind: I'm probably a really bad example, but that's probably like most of what I do. I'm a bad example. I spend my time totally interrupt driven with what customer issue has hit now. What issue has hit, what's the problem coming in and/or whine with a client and therefore I'm going to ignore all that and I'm going to be with a client for the day. In that case, I don't deal with anything but that. But then some other client's going to back up in the email inbox. I'm a bad example, but if I look at my development team, I would say it's still too much.

We don't deal with it. Many of the teams, there's too much of a requirement to deal with other things that come up, other issues that come up. Many of our teams are not to the point of being able to isolate. This group is going to handle the customer situations and the rest of them actually get to do feature work. It's hard to do, and I think one of the reasons is it's harder to do when you're building a product that you sell. It's a product you sell, not a product you run. You have a different set of issues in that kind of challenge.

In some ways, I think we care too much. I don't think you can ever care too much, but I think our focus on making sure all of our clients are successful also means the development team is more willing to jump. That customers having a problem, we're going to go solve the problem. That's a great thing, but it also means they jump too often.

Mark: One of the questions I really like to ask, because it opens up some doors, is, what's the coolest project you've ever worked on? Or the one you're most proud of? 

Rosalind: I've done a lot of unusual things at IBM, and when I was very early in my career I actually got to do common user access or CUA, our user interface standards. That actually got me into the responsibility of doing an IEEE standard for user interface as a kid. It was kind of fun, but it meant I was working with Microsoft and Sun and DEC and systems that aren't around anymore to come up with what can we do for our user interface. What can we do that is standard? My favorite one is CTRL C, CTRL X, CTRL V, because Apple had Apple C, CTRL V. IBM had CTRL-SHIFT-INSERT.

There were competing standards, and so, as part of my work, we standardized on CTRL C, CTRL V, CTRL X. In some ways, that's the most fun, because it has made a huge difference in everybody's life to have an absolute standard across Windows, Mac, everything. It's there today, so that's real.

Probably the most fun has been actually making things work in customer shops. I get to go in, deploy, make something happen, and make a change. I've done that in a lot of shops. That's been fun. Sometimes that's not fun. It's too many hours, but it's really fun in the end, because it's really there and you can see it. You can see the value. Those are my two extremes of fun.

Mark: We talked briefly last year. This is the second year that we're at DOES right now. When I talk to you next year, at this time, what do you hope to have accomplished? 

Rosalind: I hope that every single mainframe customer has started to move to modern tools and modern practices for their development and they've started the transition to servicifying, my made-up word, their back-end systems. We absolutely have to have people move.

I started in ISPF development 29 years ago, and I walk into a customer shop and the development process looks exactly the same. There's nothing on the planet that's exactly the same, except that. It's got to change. It makes no sense. That's why I did the Git work. As an IBM-er, I'm responsible for building this out of development tools, and we build Rational Team Concert. It's modern development practices for Z. It's a product we sell. It works great. We've got a number of clients who've migrated to it. It's end-to-end support.

We've got a whole bunch of clients who say, "Nah, what I've got now, it just works. Why should I move?" I wanted the Git answer to kind of say, "Okay, you're not moving because it just works, and why should you move to RTC when you're doing Git in a distributed space?" Well, I just got rid of that problem, so now what's your excuse?

Hopefully, between now and then, we'll have solved a whole bunch of the other excuses they come out with, and we'll see them moving to RTC or to Git or whatever modern SCM and modern build solution, so they can really do DevOps for the Z environment.

Mark: Is there a cherry pick somewhere people can start with?

Rosalind: Different people need to start in different places is part of the problem. Some people are just so far behind that there's just too much change, and so, in some cases, they start with a modern IDE. I will always cringe at the developers that tell me it takes more clicks to do their job now, because they have to press a mouse. Count the number of function keys they press to do the task. There are not more clicks then functions keys, but they didn't have to press a mouse before.

So, there are a lot of different places to change, and I think some customers will just start with the SCM. They can use SCM and build. A lot of them start with deployment. They try and fix the cross-platform deployment process. They start in different places. Everybody's got to start where it makes sense to start for them. Where's their biggest business value?

Mark: What time though, in this timeline of transition, do we allocate the name dinosaur on somebody who said it's hopeless?

Rosalind: Well, it's never hopeless.

Mark: Okay, let's take hopeless off the table. That it's not going to happen. 

Rosalind: If it doesn't happen, I doubt they're going to stay on Z. I think the problem that we have here is if they're going to stay on Z, because they don't have a choice, it's going to be that back-end system that they hate because they can't make changes, they don't know how to make changes, and they don't have anyone left that knows how the system works. It's just going to sit there and it's going to run and it's going to be a key critical business process and they're all going to be scared. Or they're going to transition and they're going to realize it's just a big server and it happens to be running my core business process, but it's just a big server. Kids can do development for it and anyone else can do development for it.

Mark: Is the transition going to be a generational thing?

Rosalind: It may be. It may boil down to it's going to take the next generation to make this happen, but the problem is waiting that long. That's why I say I want it this year. A lot of companies are losing the core skill that knows how their SCM and build works today. I run into many companies in which the last person in the company that knows this is going to retire in a year, so we got to move. Well, I'd rather have people move when it's not the last person in the company that knows anything about this process. In some companies, there's no one left. 

Mark: There's no corporate memory.

Rosalind: There's no one left who knows how their source code is built. They just cross their fingers it doesn't fail. This is your business running on this system. 

Mark: Is this an industry problem really that you're seeing across the industry?

Rosalind: It is a problem. In Z, it's remarkably a problem because we’ve got modules. The best thing about the mainframe is a module compiled 40 years ago will still run. The worst thing about the mainframe is a module compiled 40 years ago will still run.

It's a great platform from the standpoint of continuing, but you don't get all the value out of the latest hardware running those old things. Because we don't have the automated testing, generally, if you go into clients they haven't done the automated testing. They're afraid to change. So, if we can get the automated testing in there, then they can be more comfortable at doing more modernization, at breaking down the monolith and start making it services. If I had the automated testing, I could do that. There's lots of opportunity, but you're not going to do it. If you're in a waterfall development process and you don't do automated testing, you're not going to take the time to start splitting up the monolith because it's too risky.

Mark: Microservices and replacement of a mainframe or on the mainframe?

Rosalind:  On the mainframe. Take the existing COBOL and just split it up. You don't have to rewrite it. It provides a business function. It provides a business value. Now, some people are writing new things on the mainframe in Java. That's fine. No reason not to. It runs there just fine, so you can run Java next to COBOL. Not a problem. But you've got millions of lines of code that provide business value. Why rewrite for no value? If you're taking a section and you need to change it entirely, fine. Write it in Java and run it there. Not a problem. But if all you're doing is rewriting it to put it in Java, there's no new business value. There's no reason to do it.

We need to deal with the fact that there's that much legacy sitting there that is of that much value that people need to take advantage of.

Mark: Where does continuous integration, continuous delivery fit in the mainframe scheme?

Rosalind: Exactly the same way it fits everywhere else with a small caveat. One of the problems I have with the conference is the idea that delivery, the number of deliveries, is a measure of value.

Mark: That's Gene Kim’s premise.

Rosalind: That's the one problem I have. It took me a while to figure out where the disconnect was, but every company has a calendaring service. Every company has one. It gives them holidays. We run on the Gregorian calendar. How often do I need to change that service – a business need to change that service? Once every couple of years when a new technology comes out that I need a new interface to it. I need to add a web service, then I need to add a rest interface. Why would I deliver changes to that more frequently? There's no business value in changing that service, so what does frequency of delivery matter?

What really matters is what is the service and what is the business function you're providing. If you're providing something that's based on marketing something and you want to change the market, you obviously need to change it faster. If you've got new government regulations because of the finance industry, you've got to be able to change fast. Absolutely.

Being able to deliver fast is critical, but whether or not you have to deliver, that's based on business decision and business value. Most of the clients in the Z space are probably going to end up doing continuous delivery to pre-production, then delivery to production when the business needs it. In some cases, it may just be they get to, over time, and so I don't freak out the Z customers, not today. But they'll get to the point of maybe delivering every day. A lot of them are already delivering every week. Delivering on a frequency that makes sense for the business in smaller batches, rather than grouping up the batches for the monthly release. Grouping up for the monthly release is just a bad idea. Let's stop that.

But business value needs to be the driver of delivery. My mobile phone, I know that you're going to update the apps a bunch of times, because there's a new thing or a new advertisement or a webpage needs to change frequently. Those are going to deliver very frequently. But the back-end services, if you build an API right, why am I changing it? I need to change it for business reasons, but otherwise I don't need to do a lot of deliveries. That's why we need multi-speed IT and that's why we need to think about the fact that different things are going to go at different speeds for business reasons.