Why public-cloud-first design is the best decision an MVNO can make

Today on the podcast, I’m thrilled to be talking with Hesham Fahmy, chief information officer at TELUS. His team has taken on a huge cloud project, set big goals and is already starting to see the results. I can’t wait to talk to him about how he got the flywheel change going and the lessons learned along the way. So, let’s take 20.

Hesham Fahmy is chief information officer at TELUS. Hi, Hesham. Welcome to Telco in 20.

Hesham: [01:27] Hi, DR. Thanks for having me here.

DR: [01:28] I’m so excited to have you on our podcast. A North American friend.

Hesham: [01:33] Yes. We already got our first snowfall.

DR: [01:35] No, I think it’s scheduled to be 80 degrees in Texas, which is like 25 degrees Celsius. So anyways, I saw in the news that TELUS signed a 10-year strategic agreement with Google Cloud. So my first question as we start to talk about what TELUS is doing with the public cloud, why did you guys select Google Cloud and are you guys thinking that you’ll expand it to other vendors or going to kind of keep everything with Google Cloud?

Hesham: [02:03] Well, what I’ll say is, any technology leader knows that your long-term strategy is to be multi-cloud. So we will eventually be multi-cloud. But our goal right now is how do we get good at one cloud? How do we truly become a cloud native organization? And that’s why we decided to go first with Google. Google is going to be our premier partnership and they’re really going to be building the foundations for us. And some of the reasons why we went with Google, first and foremost, it’s a 10-year partnership that we’re doing and it’s founded on innovation. So it’s not just about moving our workloads to the cloud and us adopting cloud, but it’s where do we go beyond that and where do we co-innovate together?

And the other thing is, from their cloud platform too, they opened the door for us to also help evolve where that platform goes. So as we start adopting our workloads, we have an input into some of the product roadmaps. So that was another big factor in us deciding to go with Google. And I would say the last, but definitely not least is there was a very strong alignment to values, especially when it comes to ESG and sustainability. That really kind of aligned to where we saw the future going and who we wanted to partner with.

DR: [03:03] And that’s something in our industry, a lot of people are really fearful of the public cloud vendors. But I think one advantage that you said there is that Google has a really strong bench of talent and it’s I think very complimentary to the telco. It’s software-based cloud talent, and so it almost can work to be an extension of the TELUS team without you having to hire them or recruit these people that are some of the hardest people to recruit and pretty expensive talent. So it sounds like that was part of that strategic arrangement as well.

Hesham: [03:37] A hundred percent. And to be honest, that’s how it’s panning out. Even the bread and butter of moving our cloud workloads, that partnership, you sometimes have the teams blended in that you can’t tell who’s a Google engineer versus a TELUS engineer because we’ve worked that closely.

DR: [03:49] That’s awesome.

Hesham: [03:50] And it’s been great to leverage that talent there, for sure.

DR: [03:52] That’s amazing. And so with the 10-year strategic arrangement, tell me a little bit more about the long-term vision. Is it really about modernizing your workloads? Is it a strong vector around cost savings and efficiency? Talk to me a little bit about what you guys are trying to do with the cloud at TELUS.

Hesham: [04:10] It’s really a combination of both the things you said, but if I step back a little bit, the 10-year vision is, we want to be firmly a technology company that happens to be a CSP at the same time. But we really want to be truly software centric, everything software defined in building a platform that enables many second and third order offerings to our customers. And so to do that, we need to really build a strong technology foundation, but then how do we make sure a lot of our engineers’ time now is spent on building these value add for our customers and not spending it on managing infrastructure, managing day-to-day operations? And so that’s part of our bet to go into the cloud. And then when you said is it around moving workloads, cost savings? It’s really a bit of both, but when I look at it, it’s what opportunities does the cloud unlock for us?

Do all of a sudden our time to market decreases drastically? Are we able to innovate a lot quickly? Are we able to experiment a lot quickly and not have the tax of managing infrastructure or procuring infrastructure? I think that’s what we like about the cloud is, we want to move those workloads, that agility and experimentation that it can bring to the table for us. And then I think when we look at the cost, there’s definitely a cost play there as we start to exit some of our data centers. But I think we all know that’s usually the long tail. Usually in between, you have a little bit of spike of costs when you’re in the hybrid mode when you’re operating both environments.

So when we look at cost, we look at the total cost of ownership, which is not just the cost of running infrastructure, but what’s your true cost of getting ideas to production? And so what’s the cost of actually you software development? What’s the cost of your testing? What’s the cost of managing environments? And so those all factor into the costs and you can start seeing immediate savings that we get there as well.

DR: [05:46] Like you said, it’s a journey and it would be nice to be like Samantha, Bewitched, and snap your fingers and all of your workloads are magically refactored, running efficiently. But that’s not how the cloud works, like you’ve mentioned. I call it the double bubble where you’re paying for the production workload on premise or wherever it existed before while you’re building the new and that costs money. And so there’s this period of time where you’re paying for it in both places and that makes people really nervous.

Where have you guys decided to start? Are you starting with network workloads? Are you starting with BSS workloads? Are you starting with non-production workloads to build up that familiarity with how the cloud works?

Hesham: [06:27] So I’m going to have a strange answer for you, but we’re talking all workloads and the reason is because we have some very aggressive targets. Our goal is to be 80% of all our systems and that’s everything from IT, BSS, OSS network to be in the cloud by 2025.

DR: [06:41] That’s awesome. And that’s not a lot of runway.

Hesham: [06:43] No.

DR: [06:43] So we can’t dabble with small workloads, and instead we really need to spread the brush really wide. And so in choosing what we’d go first, we did an inventory of all the applications and then we just started creating a cloud disposition for each and saying, “Okay, for each application, what is its most likely journey? Is this an application that’s ripe for some serious refactoring? And if we did the refactoring, what would it take? And then do you wait until all the refactoring done or do you do interim steps of some lift and shift in the way?” And then other apps, they may not have any destination to be refactored, they’re so legacy, you’re probably going to get most of your bang for your buck just lifting and shifting them. So we did that kind of pass across all the applications and built almost like if you think of a heat map and then we’re just ruthlessly now tackling that heat map. And so whatever comes first in the heat map, we’re moving those first.

And are you doing easiest first by area?

Hesham: [07:34] We probably started the first maybe six to eight months doing the easiest first, but we were hitting a roadblock where we’re getting a lot of diminishing returns where velocity is slowing down because there’s no more of those easy applications. But we haven’t tackled some of the big rocks that we had, which is how do we handle databases in migrating databases? We have a big web logic footprint. How do we move web logic to the cloud? And so what we ended up pivoting earlier this year is saying, “No, we’re going to actually tackle some of the big ones.”

Because if you solve one or two of the big ones, you probably unlock a lot of the big problems and solve them and build the patterns that brings the masses behind. So that’s we pivot, we call them icon apps too. So then we try saying, “Hey, let’s pick icon apps that both are complex so we can solve the complex problems and unlock it for other apps to go but also put them in the cloud and start getting some quick wins that our business can benefit from.” And now we start getting tailwind of goodwill that everybody’s seeing what cloud can do for the business.

DR: [08:25] I think that’s a great point. Yes, it’s great for the business, that’s why we all exist and we have jobs is to benefit the business. But I think, by focusing on iconic apps, which I love that name, I’m going to use it a ton, by focusing on these big boulders, you’re doing a second thing for your employee population. When you guys get an iconic app across the line that it’s working in the cloud, it’s more efficient, it might even be easier to manage. You start to change the hearts and minds of people who are skeptical, doubtful, or maybe even against the move and they start to realize, “Hey, this might be okay and it might be even good.” And that starts to help that flywheel of change within your organization where people get energized to move the next iconic app like, “Oh that was so easy, we can do that again.” And they start to ideate and you start to build that energy and momentum, which you totally need when you’re going to move 80% of your workloads by 2025.

Hesham: [09:23] A hundred percent. And we saw that, like you said, it’s that inertia that builds up. I’ll tell you a funny story. When we did that pivot early in the year, like I said, and we’re going to focus on the big rocks and the big boulders, we created these, what we called them release trains. And so we would say, “Hey, here’s a release train where we’re going to try to get X number of apps into the cloud.” Think of it like your ticket on a flight. And we told all the application teams, “Now here’s your blueprint, here’s your decision tree, here’s a lot of the help that you’ll get, now start booking your tickets.” People were scared, people did not want to do it. For the first three or four, we had to force application teams to go in, like “This was not a choice, you are going to do it.”

DR: [09:56] Here is your ticket, get on the train.

Hesham: [09:58] Exactly. Here’s your ticket and you cannot miss your flight, man. And so we did that. It was a huge success. Obviously, the first few trains, that’s where you had the bumps and scars and we learned through it. But coming to the third, fourth, fifth release train, they became very smooth. And then a lot of teams that went through it, first of all realized it wasn’t as painful as they thought. And two, coming out the other side, to your point, things got a lot easier for them. So managing their own infrastructure got a lot easy. Managing when they have incidents and having visibility for what’s going on really kind of unlocked a lot of them. And the word of mouth started spreading that this was good for us and so much so now that I think we’re on release train maybe 15 or 16 right now, and we’re having to turn away applications because they’re oversubscribed.

DR: [10:37] That’s awesome. And what’s so important about those first couple of release trains is I imagine you needed to publicize the wins. There were some bumps in issues, but here we are. And you just have to announce it because the type of people that get attracted into engineering roles, don’t really toot their own horns. And you have to do that PR to convince the masses that this is working, we’re doing it, it’s inevitable and people are seeing success with it because it gets those people to jump on the trains. But it sounds like you’ve done that work to publicize the wins. So now, people are not afraid of it and they’re like jumping on the train. So that is fabulous. That’s awesome news.

Hesham: [11:16] And it’s really good that you call that out, DR, because a lot of people think that this journey is all about the technology, it’s all about understanding cloud, it’s understanding cloud native.

DR: [11:25] Exactly.

Hesham: [11:26] But actually there’s a huge PR piece to it and it was something that we really overlooked. But then when we started double down on that, both internal marketing to our own engineering teams and having them speak and vouch for what’s happening, but also like I said, going back to our business, we’re in the job of delivering value to the business, but sometimes they don’t see that value. But then when you start highlighting, “Wait a second, our performance just improved by two x. Well why did that happen? It wasn’t magic. It’s because of some of this modernization we did.” Or when we talk about our meantime between failures dropping drastically, again, we’re not sort of tooting our horns and saying why and how. And so that’s been a big PR machine that we’ve driven as well to make everybody understand that this has come as part of this whole vision and cloud journey we’re on.

DR: [12:06] I think you guys announced the Google deal either in ’20 or ’21. This is a four or five year journey and you have to remind people, “We’re not changing our mind, this is where we’re going and it’s working.” Something else that you said a few minutes ago was kind of giving people these patterns for refactoring and rearchitecting and how you’re moving those applications. So it sounds like you guys are refactoring your applications, which is exactly the way you should be using the public cloud. I talk about it doesn’t make sense to use a public cloud if you’re buying a new house. You wouldn’t move in and not unpack the boxes. You would use the house, you would set up your bed and sleep in the bed and all that stuff. And so to use the public cloud well is to use the Google cloud databases, the Google services and starting to change your legacy applications. And so, are you guys doing that or are you trying to keep things cloud agnostic so you can move them to another cloud or back on premise later?

Hesham: [13:03] So that’s a big question. I need to unpack that in a couple layers here. Maybe I’ll answer the last part you said first, which was, are we trying to keep workloads agnostic? That is definitely what we want to do. Not so much that we want to bring things back on premise, but we actually want to take the cognitive load from a development team to actually worry about where does this get deployed? We want them focused more on your application and its logic and then let the deployment of where it needs to be located as a workload be determined later on made by how the application’s used and make that not be something I think of up front. And if you think again, where the industry’s going, because we’re going mobile edge compute as well and everything, that notion of call it your data center becomes very federated. And so we definitely are on board on that.

And so, we’re making our bets on Kubernetes being the container orchestrator, Anthos being the control pane that makes it go across then whether it’s Google Cloud or other clouds or [inaudible 00:13:56] computer on prem. So that’s definitely the approach that we want to take. But going back to what you said, are we refactoring or doing maybe what you would consider a lift and shift? Again, I would say it’s a spectrum. One of the things we talk about lot and I tell my team is, one, don’t think of everything as a one size fits all. We have such a heterogeneous environment that there’s never going to be one solution that works for all. And then the second thing I also tell them a lot is, fall in love with the problem and not the solution. Because a lot of times we get fixated on, “oh great, I want to use serverless and I love serverless or I want to use pubs up.”

DR: [14:29] That’s forcing.

Hesham: [14:29] And you start forcing it versus instead focus on what problem do you need to solve in your application where you go and then look for the solutions for it. Ideally, like you said, if we’re going to get the maximum value out of cloud, we need to refactor. But even refactoring itself has a continuum. Maybe the first thing I just look at refactoring is making my application stateless so that I could put them in a Kubernetes environment, but then they can truly autoscale both up and down, but also shift the workloads. But you can’t do that if you have a very stateful application. So that’s level one of refractory.

Level two is going to databases and then refactoring more of the cloud native databases, cloud [inaudible 00:15:05] on GCP or you change your database and do you need a relation database or do you go a key value store? So that would call level two refactoring. And then I would say level three refactoring saying, “Well do you actually even look at your whole logic and do you really want to be event driven?” And so that’s a spectrum too. And again, you don’t want to start at level three, you might start at level one or level two.

DR: [15:23] And then come back to it later.

Hesham: [15:25] Exactly. And it becomes part of a journey you do, right? Because if you wait to just do a level three, then you’re probably going to disappear for maybe a whole year, a year and a half before you show any value. Versus if you do the level one first and the level two, you’re delivering value that’s demonstrable every couple of months. And that’s very important for us too. But the last thing too is not everything has to be refactored. There are some workloads that say the juice isn’t worth the squeeze trying to refactor them. And you probably still can get a lot of benefits.

A lot of people write off what you can get from doing a lift and shift, but even if you took them as straight VMs and moved them over and did them as Google compute, there is still some flexibility you get there because one of the biggest problems most enterprises have is managing those many test environments and that they start drifting in configuration from production. And so it’s a long-winded answer to your question, but it’s all of the above. It’s refining the right fit for purpose path, if you will, for each application we have.

DR: [16:16] Well, I think what telco has to get comfortable with, especially, and I don’t think it’s really been part of the DNA thus far, is understanding how dynamic the cloud environment is versus the way that we’ve been running workloads to date. Things are constantly changing in the cloud and so you have to think in a way where everything’s changing all the time. Databases are being upgraded, workloads are moving into production. And so the dynamicism of code and change is incredibly high. The operational runbooks have changed, like you said with the testing environments

Hesham: [16:51] A hundred percent. And I think that’s the hardest thing for a lot of people who’ve been running the traditional way to wrap their minds around is that dynamicity of the environments. What cloud allows you to do is really shrink the blast radius of what you need to do with changes

DR: [17:04] Exactly.

Hesham: [17:05] If you get something wrong in a traditional legacy environment when you’re doing anything on-prem, those become very costly mistakes because to unwind that-

DR: [17:12] Because they go to the whole world.

Hesham: [17:15] The whole environment goes down or to roll it back, it’s a whole deployment that will take many hours and so it becomes very expensive. But when you start adopting some of the cloud native techniques, you can actually make those very cheap.

DR: [17:24] We use canaries, right? We’ll say, “Okay, let’s just try this small change on a friendly group.” “Oh that didn’t work.” And you haven’t really, like you said, a huge blast radius, you haven’t impacted the entire environment. It’s fantastic. So speaking of dynamicism, I live in Austin, Texas, which is home to Circuit of the Americas F1 track. It’s a famous track, it hosts tons of different races and you were just here racing your motorcycle.

Hesham: [17:49] I was. I was. Yes. Yes.

DR: [17:50] This track is famous for two things. It has the most left turns of any F1 track. Everyone has to train the other side of their body. And then number two, turn one is a steep incline with I think a hairpin turn. So how did turn one treat you?

Hesham: [18:04] Oh, I love that turn. It’s actually one of my favorite turns on the whole track because first you need absolute commitment because you’re going uphill and you can’t see it until you crest at the top. And so you just have to know this is where I point my bike to and trust that that’s going to give me the right place.

DR: [18:17] It’s kind of like the cloud.

Hesham: [18:20] So it really teaches you that commitment. And then the other thing I love about it, because it’s at the end of a front strait, and so you’re clearing easily 200, 203 kilometers an hour and you think that you have to break a lot earlier, but because it climbs so far uphill that uphill slows you down. I’m still trying to learn it but I’ve see some of my fellow riders and some of the pros and they’ll break so deep in itm it’s mind boggling how they can do it. But the physics of going uphill.

DR: [18:43] No, Louis Hamilton was like exactly what you said, timing the apex of the turn is interesting and hard. Well I think you’ve shared a little bit of a video from your helmet camp and said we will put that in the show notes because that is badass.

Hesham: [18:58] Oh, wow.

DR: [18:58] Totally awesome. And I think you were going like 243 kilometers per hour.

Hesham: [19:02] I’ll trust you for that. But yes, I think so.

DR: [19:03] Yes. I think that’s like 150 miles per hour for Americans.

Hesham Fahmy: [19:06] Yes.

DR: [19:07] So that was awesome. Now Hesham, this was a great conversation. I think I could have talked to you for another 30 minutes, but it sounds like TELUS is making great progress. You have ambitious goals. That flywheel of change is starting to turn and I wish you the best of luck in your move to the cloud.

Hesham: [19:24] Thank you so much, DR.

DR: [19:24] Awesome.

Hesham Fahmy: [19:25] It’s been a real pleasure chatting with you.

DR: [19:27] Stick around because we’re ending each podcast with a Telco in 20 takeaway. I have 20 seconds to tell you something you need to know.

I really like Hesham’s framework for refactoring. It makes sense because moving to the cloud is not one and done, it’s a continuous evolution of your workloads. Your teams need to be prepared to change from managing a large and static environment to managing a more dynamic one where everything from the machine to the app is changing on the regular. Let me repeat his simple framework. Start by taking an inventory of your applications to figure out which ones are right for refactoring, then break them into levels. Level one is quick and easy stuff like resizing machines and rethinking capacity needs. Level two were things that need deeper decision-making like refactoring databases.

Level three is where you bring out your jackhammers. These are bigger projects on what Hesham calls iconic apps that will take a longer time to refactor but successfully transforming them can change the economics of running it in the public cloud dramatically. So you have to bite the bullet and get it done. And as a side benefit, your team will learn things along the way that will give them ideas for future cloud projects. And that’s it, 1, 2, 3, and you’re on your way. Pretty soon, your flywheel of change is spinning fast. Refactoring is the key to making the public cloud sing.

You know what else makes angels sing? This podcast. Be sure to check out our other heavenly episodes. Don’t forget to share them with your colleagues. Follow us on Apple Podcasts and Spotify and leave us a review. DM me on Twitter because Elon hasn’t ruined it yet @telcoDR. Connect with me on LinkedIn. Sign up for our awesome email newsletter on telcodr.com and check out our killer YouTube channel. And join me at MV News Summit at this year’s MWC in fabulous Barcelona. Totogi is a gold sponsor. I’ll be talking and we’ll be throwing a party you can’t miss. Later, nerds.