Talking AI with Telecoms.com

DR: [00:00:02] Yeah.

Scott Bicheno: [00:00:03] Simple headline, said, “Totogi automates BSS code writing.” And your claim is that you were going to build 500,000 lines of BSS code in one day, and we’re-

Iain Morris: [00:00:13] So this is what you’ve just done?

Scott Bicheno: [00:00:13] This is what they’ve just done, yeah, yeah.

DR: [00:00:13] Yeah, I mean. We really started with the premise of no one believes us, the CSPs are skeptical. And I think there is a lot of AI slop out there.

Scott Bicheno: [00:00:20] Yeah, it’s attached to everything right now, isn’t it?

DR: [00:00:22] How do you prove it works? And we’re like, “We’re going to bring a coder and like, code in front of you.”

Scott Bicheno: [00:00:28] Yeah.

Announcer: [00:00:33] This is “Telco in 20,” a podcast that helps telco execs achieve a competitive advantage with AI and the public cloud. It is hosted by Danielle Rios, also known as DR. Today we’re talking with the hosts of the Telecoms.com podcast, Scott Bicheno and Iain Morris.

DR: [00:00:53] Hi guys, I’m DR.

[00:00:55] At the end of October, Totogi did something unprecedented at Telecom TV’s AI-Native Telco Forum in Germany. We created six BSS modules from scratch with AI, using Totogi’s BSS Magic and one forward-deployed engineer.

[00:01:10] I knew there were going to be a lot of questions about it, so I made a pit stop in London to talk to the Telecoms.com podcast firing squad of Scott Bicheno and Iain Morris. Why? Because a big part of being an evangelist is debating with the non-believers.

[00:01:25] Scott and Iain are smart, funny and ready to call out anything they smell as BS. Last month when AWS had that massive outage in US-EAST-1, Scott was practically salivating at the opportunity to grill me about it. When Totogi generated more than 500,000 lines of working code live at the Telecom TV event, Iain wanted to know how the ontology really works. So I picked up a case of their favorite beer, nothing like a good bribe to butter them up, and joined them in their studio.

[00:01:55] I’m bringing our conversation to you in this special episode of “Telco in 20,” which is also one of our rare double-drops. The Telecoms.com team and I are both sharing it on our podcast channels. We talked about Totogi’s feat, if AI is really all that evil, and of course touch on the resilience of the public cloud. So, grab a brewski, settle in and let’s take 100.

Scott Bicheno: [00:02:25] Hello and welcome to your Telecoms.com podcast of the week. And I’m delighted to say we’ve got a guest again, a returning guest, Danielle Rios. I nearly said Royston.

DR: [00:02:37] I know. I know. It’d be easy, because it’s an English name. But…

Scott Bicheno: [00:02:37] Well at least it’s DR all the way.

DR: [00:02:38] It still is, yeah.

Scott Bicheno: [00:02:39] Often known as… You often go as DR, don’t you?

DR: [00:02:40] Always, yeah, yeah.

Scott Bicheno: [00:02:40] Oh, always, okay. Well I’ll just call you DR then. Like JR, in Dallas.

DR: [00:02:45] Yeah, like that. Yeah.

Scott Bicheno: [00:02:46] Yeah. Welcome back Danielle, or DR.

DR: [00:02:48] Yeah.

Scott Bicheno: [00:02:49] No, I’ve got to call you Danielle.

Iain Morris: [00:02:50] We’re probably the only two people who call you Danielle.

DR: [00:02:52] Oh, probably. Yeah, you might be. But my one question is, if I come on five times, is there like a five timers jacket like on SNL?

Iain Morris: [00:02:58] Oh, we should do that. That’s a good idea.

Scott Bicheno: [00:03:00] No, we don’t have any merch.

DR: [00:03:02] No swag?

Iain Morris: [00:03:03] We need merch.

DR: [00:03:03] It’s more like a trophy.

Scott Bicheno: [00:03:05] We should give everyone a metal T-shirt or something.

DR: [00:03:07] Totally.

Scott Bicheno: [00:03:08] For people who don’t know, this Yeti that we use every single episode, Danielle brought us…

Iain Morris: [00:03:13] It was after the pandemic.

DR: [00:03:14] ’22?

Iain Morris: [00:03:14] ’21.

Scott Bicheno: [00:03:15] Oh, it was post-pandemic, was it? God, I thought it was pre. There we go.

Iain Morris: [00:03:17] No. She’d already done the taking over the Ericsson stand at Mobile World Congress by then.

DR: [00:03:21] I had done Cloud City.

Scott Bicheno: [00:03:21] Right. Okay. Well we were having a go at public cloud, which we might do again today.

DR: [00:03:25] That’s fine.

Scott Bicheno: [00:03:26] Given what happened earlier on this week.

DR: [00:03:27] Sure.

Scott Bicheno: [00:03:27] And then you leant into that, we called you a public cloud cheerleader so you brought pom-poms and stuff like that.

DR: [00:03:32] I did.

Scott Bicheno: [00:03:32] Which was a cool thing.

DR: [00:03:32] And number one, a giant foam number one.

Scott Bicheno: [00:03:34] Oh yes, that’s right, one of those pointy fingers. And then we had you on as one of our guests on the Mobile World Conference, MWC one.

DR: [00:03:41] Which was a great idea.

Scott Bicheno: [00:03:42] You brought two cocktail waiters and a trolley, and they were mixing us old-fashioneds live while we were recording. I remember the first sip of it, because of course, being us, we’d been out a fair bit prior to that. First sip of that I was like, “Ooh.” And then it kicks in and then… suddenly, “Right, I’m ready to rock.”

Iain Morris: [00:04:00] It’s like medicine, then. It was medicinal, yeah.

Scott Bicheno: [00:04:01] Yeah, exactly.

DR: [00:04:02] Yeah, it’s hair of the dog maybe a little bit. It was delicious.

Iain Morris: [00:04:04] It was good, very good.

Scott Bicheno: [00:04:05] So I’m going to say, actually, you’re hiding something under…

DR: [00:04:08] I am hiding. I always bring gifts. I sort of feel like, I don’t know, the hostess in me.

Scott Bicheno: [00:04:12] So I thought I’d let you get that sorted.

DR: [00:04:15] I noticed you guys like this BrewDog.

Scott Bicheno: [00:04:20] Oh wow, look at that.

DR: [00:04:20] But the Yeti koozie will replace the janky koozies you have.

Iain Morris: [00:04:23] It’s got your brand on it.

DR: [00:04:26] You know me.

Scott Bicheno: [00:04:27] Branded.

DR: [00:04:27] So there you go.

Scott Bicheno: [00:04:27] That is very cool. And you even put…

DR: [00:04:29] I put a BrewDog in there for you guys.

Scott Bicheno: [00:04:31] Excellent.

DR: [00:04:31] Here’s one for Iain.

Iain Morris: [00:04:33] Thank you very much.

DR: [00:04:33] There you go.

Scott Bicheno: [00:04:33] So we’ll swap out these BrewDogs for…

Iain Morris: [00:04:33] Yes.

DR: [00:04:36] And then I brought one for Pierre.

Pierre: [00:04:38] Oh, you didn’t forget me. Thank you.

DR: [00:04:39] Yes. And then I branded with me…

Iain Morris: [00:04:41] Thank you very much.

Scott Bicheno: [00:04:42] And you’ve got your own one.

DR: [00:04:43] TelcoDR.

Iain Morris: [00:04:44] Nice.

Scott Bicheno: [00:04:46] Oh, nice one. Your attention to detail is unsurpassed, Danielle. Thank you very much.

DR: [00:04:51] Well, I have a really great support team that helps me do amazing…

Scott Bicheno: [00:04:55] They do. And we should say, we’ve got your right-hand person, Lindsay…

DR: [00:04:58] Yes, Lindsay is off-camera.

Scott Bicheno: [00:04:59] … in the background. Hi, Lindsay.

Pierre: [00:05:00] And the beer is cold.

Scott Bicheno: [00:05:01] And the beer is cold.

Iain Morris: [00:05:02] It’s cold.

Scott Bicheno: [00:05:02] Good skills. And we should also give a shout out to Jen Hibberd from Liberty who works with Elena. And she… Unfortunately we only got room for one plus-one in here, so she’s chilling or taking in the view or whatever outside.

DR: [00:05:12] Well she’s minding the beers. No, we have extra beers here.

Pierre: [00:05:15] Oh my God.

Scott Bicheno: [00:05:16] She’s got a whole suitcase of beers.

Iain Morris: [00:05:17] That’s lots of beer.

DR: [00:05:18] I mean, I know Pierre only has a couple of hours, but…

Pierre: [00:05:20] There’s a green one too.

Iain Morris: [00:05:23] Oh my God. She’s basically, for people listening and watching, she’s bought a s**tload of BrewDog.

DR: [00:05:26] 2, 4, 6, 8, 10, 12, 14, 16 beers. For four of us, I am not going to have four beers on this podcast.

Iain Morris: [00:05:33] We’ll be all right today, then.

DR: [00:05:33] Do you want a couple?

Iain Morris: [00:05:36] Yeah, just grab a couple.

DR: [00:05:36] Here you go.

Scott Bicheno: [00:05:37] Nice one. Thank you very much, Danielle.

Iain Morris: [00:05:38] Thank you.

DR: [00:05:39] You’re welcome.

Scott Bicheno: [00:05:40] And I love the branding. This is very cool.

DR: [00:05:41] Yeah.

Scott Bicheno: [00:05:42] So we’ll crack on, because we’ve got a time limit. It’s great to have you here. You’ve made a big effort, you’ve just been in Germany, haven’t you?

DR: [00:05:48] I was just at the new AI-Native Forum that Telecom TV put on.

Scott Bicheno: [00:05:53] Yeah, with Ray and co.

DR: [00:05:53] Yeah.

Scott Bicheno: [00:05:54] It sounds like, judging by LinkedIn, it sold out as well. So they’ve done well there.

DR: [00:05:57] It was.

Scott Bicheno: [00:05:57] Good for them.

DR: [00:05:59] They had to rearrange their floor plan to make more room for more people, which was great.

Scott Bicheno: [00:06:03] We should buy them, Iain.

Iain Morris: [00:06:04] We should, yeah.

Scott Bicheno: [00:06:05] We’ve bought Ray before, we can do it again. Why don’t we buy Steve while we’re at it, what can go wrong? Don’t worry, Steve doesn’t listen to this.

[00:06:14] But I’ll get straight into it then, Danielle, a possible to you. What were you in Germany at Ray’s event for?

DR: [00:06:20] Yeah. So they really wanted that event to focus on real use cases. How are operators really getting AI into their organizations? It is a struggle. I mean, it’s not a telecoms problem, it’s every large organization. I mean, there’s kind of maybe three pieces. There’s the technology itself, what tools do you pick? There’s training your people, because no one really knows this. And then I think there’s a culture shift, because there’s a natural reaction to fight it or resist it or question what this means for me.

Scott Bicheno: [00:06:53] I bet you saw it a few months ago, I think it was, we spoke about it on one pod where it was just me and Iain and we announced at the start that we were going to s**t on AI. And there was a study out of MIT or somewhere, that was talking about a very high proportion of…

DR: [00:07:06] 95%

Scott Bicheno: [00:07:07] Yeah, of enterprise implementations weren’t paying off. But it wasn’t… It was nuanced. It wasn’t saying it’s because AI is rubbish, it’s because the implementation wasn’t done right.

DR: [00:07:16] Yeah. Like I said, I think you really need to think through the HR side, the human side of rolling it out. And I think there’s a lot more than to just putting together an agent and throwing it out there, I think you need to really think through what’s called context engineering. How do you give the AI the guardrails, the context of the work, so-

Scott Bicheno: [00:07:36] Right. Sounds like an escalation of prompt engineering. It’s like giving people-

DR: [00:07:38] Yeah, it’s a little bit more than prompt I think is kind of where we’re at. But where you give it enough of guardrails so that the answers are more likely to be correct. And what I mean by more likely, like, high nineties. And so that you can let it loose into your organization and feel like it’s not going to ruin your business or…

Scott Bicheno: [00:07:54] Is even that high enough? Or are we getting into that four nines or whatever they call it?

Iain Morris: [00:07:58] Five nines.

Scott Bicheno: [00:07:59] Five nines.

DR: [00:07:59] Yeah.

Iain Morris: [00:08:00] That’s a reliability thing for carrier infrastructure really, you know.

Scott Bicheno: [00:08:03] I know, but if you are going to, to quote Danielle, let it loose on your business, I mean is even a 5% error rate too high?

Iain Morris: [00:08:11] But that’s downtime for a network, whereas if you’re-

DR: [00:08:14] We’re just-

Scott Bicheno: [00:08:15] I’m not saying exactly the same, but I’m querying, I’m setting up a whole new query of whether or not we need to aim for a similar level in terms of… And this might be relevant when we bring up the AWS thing, in terms of if it even goes wrong 5% of the time. Isn’t that bad?

Iain Morris: [00:08:28] Oh, but that’s different. That’s not an AI thing, that’s just an out-

Scott Bicheno: [00:08:30] No, but it’s downtime or it’s things going wrong.

Iain Morris: [00:08:34] Yeah. But-

DR: [00:08:35] Or just correctness, right? I think he’s asking how correct does it need to be? And I think, again-

Iain Morris: [00:08:38] Yeah. I mean, that’s mistakes rather than… Because sometimes it doesn’t need to be on, the AI is not doing anything. You see what I mean?

Scott Bicheno: [00:08:43] That’s true. And the mistakes don’t need to be catastrophic. Yeah, I get it. But they could be.

DR: [00:08:45] Well, I think it depends on where it’s deployed. I think people will handle a mistake in a chatbot if you can pull an andon cord or whatever it’s called, like on a bus and say, “Escalate me to human.” Right? It’s not that big a… You can say at the bottom, “Disclaimer: this is AI and it may make mistakes.” But I think again, in a network you might need higher correctness.

Scott Bicheno: [00:09:05] No, totally. Iain is right to correct me on that. I wasn’t making a direct connection, it just made me think of these ways of measuring accuracy and perfection.

Iain Morris: [00:09:12] I mean just on that, though, on the kind of… They call it hallucinations, don’t they? Which I think is a bit of a strange expression for it, because it’s just like mistakes to me.

[00:09:20] But I noticed even… The one that I encounter all the time, because you can’t really escape from it actually, is Gemini on Google. Because you’re going to use Google as your search engine. And unfortunately these days there’s sort of no way of deactivating their AI. And it comes up with these summaries at the top of anything you put in, and they’re frequently really quite badly wrong, even on quite simple queries sometimes. I remember putting something in just to test it, on countries with a population of fewer than 5 million people. And it was coming out with things like South Korea, which is like, I don’t know, 40 million or something.

[00:09:54] And I just kind of wonder what-

Scott Bicheno: [00:09:56] And did it have a hyperlink? Because it normally has a link to the source.

Iain Morris: [00:09:59] I can’t remember.

DR: [00:10:00] It usually does in… If you’re searching on Google at the top and you’re talking that top summary, you can say like, “Work more with AI,” and it’ll give you the source list.

Scott Bicheno: [00:10:08] But even if the source was errant, it’s still not very clever of it to cite it, is it?

Iain Morris: [00:10:11] But the other thing I’ve noticed people saying about generative AI is it’s good for… which I found really curious about AI, because it’s sort of contrary to what I’d expect, is that it’s very good for getting it to write, say, a paragraph of text. I mean, it’s not going to be Martin Amis, but it can generate something that’s workable. But if you put simple arithmetic into it, sometimes, it makes mistakes. Whereas I would’ve thought that AI would be… That’s what it would be really good at, is the number stuff. It’s just machines and figuring…

DR: [00:10:37] Yeah. There’s all these different packages that you can use with the LLM you’re working with. I’d say the two main models that everyone really uses is the OpenAI one, which is known as ChatGPT, and then the other big one is Anthropic’s Claude, that a lot of people haven’t used. Those are probably the two best.

[00:10:53] Elon Musk is working really hard with his Grok one, and it’s coming up very strong. Surprisingly, he’s done it. DeepSeek from China, famously trained and took a lot less dollars to train it to become really good.

Scott Bicheno: [00:11:08] A lot less overhead, yeah.

DR: [00:11:08] But I think the two main ones are OpenAI and Anthropic. Those are the two kind of neck-and-neck ones.

Iain Morris: [00:11:13] Are you… because the DeepSeek story has been quite controversial, isn’t it? It’s like, did they actually really get access to chips they weren’t supposed to? Is it as efficient as it’s cracked up to be?

DR: [00:11:21] I don’t know if… Yeah.

Iain Morris: [00:11:21] I’m not sure where that’s gone since the…

Scott Bicheno: [00:11:21] … pod we did earlier on this week.

DR: [00:11:24] Yeah, I don’t know whatever happened. Supposedly they kind of went through Singapore or something, and got their hands on some NVIDIA chips or whatever.

[00:11:31] But I think there’s definitely model differences. Like with Gemini, you notice I didn’t put it in the top three. I think it is way off.

Scott Bicheno: [00:11:38] That’s interesting.

DR: [00:11:39] I think with math, there’s components you can call to make the math more reliable, just like there’s different tools you would use for coding. There’s different tools for picture generation.

Scott Bicheno: [00:11:50] Claude’s considered better for coding, isn’t it?

DR: [00:11:52] Yeah, it still is.

Scott Bicheno: [00:11:54] The Anthropic one.

DR: [00:11:54] ChatGPT is really trying to challenge them on that. Their latest drop that happened at the end of September, they were like, “We’re as good as, and maybe even better.” But then when engineers got their hands on it they were like, “Claude’s still better.”

[00:12:05] But the more you use it, the more you realize how you can make it be more right. And some of that is you as a prompter becoming skilled. And it’s just kind of like how you had to become a good searcher on Google. If you didn’t know how to use keywords and do it right you can get kind of crappy results, and you’re hunting for what you’re looking for. But if you get really good at Google, then usually in the first page you’re going to get your answer. And the same thing is with prompting.

[00:12:31] And then the second thing is really learning about giving it the context it needs so that it’s more likely to produce the right answer.

Iain Morris: [00:12:39] So the prompting, just to be clear, because a lot of people…

DR: [00:12:41] Don’t know?

Iain Morris: [00:12:42] Yeah, and they do it in a casual way like, “Tell me this or that.” And I know Scott’s son probably knows more about this stuff, because I think he’s been doing some of that.

Scott Bicheno: [00:12:49] He’s been studying, yeah. Prompt engineering is a big thing.

Iain Morris: [00:12:51] But prompting would be natural language, but just being very specific about what you want, essentially, and putting in the right kind of expressions and being very, very clear about what you’re looking for.

DR: [00:13:00] Correct. You kind of want to bound it a little bit. “I want my tone to be this. My audience is this, the length is this, the topic is this. Don’t talk about that.” Because if you just say, “Write a paragraph on the cloud…”

Scott Bicheno: [00:13:14] It’s got too much to choose from, isn’t it?

DR: [00:13:15] It doesn’t know if you’re private, public, AWS, Google.

Scott Bicheno: [00:13:20] Does it ever come back and go, “Can you be a bit more f**king specific?”

DR: [00:13:22] It does. It does. Like sometimes by accident I hit enter, I’m like, “Let’s write a blog,” and I hit enter.

Scott Bicheno: [00:13:29] Right, and it goes.

DR: [00:13:29] And it’s like, ” About what?”

Scott Bicheno: [00:13:30] “Help me to help you.” Yeah.

DR: [00:13:32] Right? Or I’ll be too high-level and it’ll say, “Okay, if you give me some more instructions I can write a really great blog.”

[00:13:40] But both of those products have the concept of a project, where it’s like, “Okay…” Like, I have a blog project and I have custom instructions that’s like, “This is always for a telco executive. Make it about 1200 words long. Your tone should be this. Always start with a controversial take that…” Whatever. And then, “Cite examples. Please research CSP examples. Don’t use a bank.” Right? “Use Vodafone,” or whatever.

Scott Bicheno: [00:14:08] Yeah. I mean, I suppose… I don’t use AI very much at all. I know that’s quite heretical of me in this day and age. But I would imagine what would be useful, especially to someone who hasn’t bothered to learn prompt best practice, as you seem to have, is to have a drilling down process. Like, “Scott, you asked for this. I can give you a go there, but you might want to…” Or you suggest parameters, suggest prompts maybe.

[00:14:28] But then I suppose that’s a bit circular, isn’t it? You don’t want the AI suggesting prompts to you, they should be yours.

DR: [00:14:36] No, it can.

Scott Bicheno: [00:14:36] So it can get a bit circular.

DR: [00:14:36] It’s called meta-prompting.

Scott Bicheno: [00:14:38] There we go.

DR: [00:14:38] So let’s say you work with it and you finally get to something… You’re like, “Oh, that was actually what I was looking for. That’s the paragraph that I wanted.” What I’ll do is I’ll say, “Write the prompt that would’ve produced this much sooner in our dialogue.” Boom. And then you save it.

Scott Bicheno: [00:14:52] Right, I see.

DR: [00:14:57] And then you save it.

Scott Bicheno: [00:14:57] Oh, and then you know for next time.

DR: [00:14:57] Yeah, put it in a doc and be like…

Iain Morris: [00:14:57] It’s trial-and-error.

Scott Bicheno: [00:14:58] No, that’s a good idea.

DR: [00:14:58] Yeah.

Scott Bicheno: [00:14:59] No, but that’s right. Reverse engineering it to say, “What could I have said in much-

DR: [00:15:02] I’m a dumbass, but write the prompt and then-

Scott Bicheno: [00:15:04] What could I have said in much fewer words to get this design outcome?

Iain Morris: [00:15:07] Yeah. You can reverse-engineer, but it’s trial and error to some extent. And then you get it right, and then you save that and then I guess you can refine that in future to make it even more specific.

Scott Bicheno: [00:15:14] It’ll be a whole skill set for everyone to learn.

DR: [00:15:15] Absolutely. Right? You can make them part of the instruction-

Scott Bicheno: [00:15:18] Well, perhaps not everyone. Except Luddites.

DR: [00:15:20] Well, everyone I know that’s been-

Scott Bicheno: [00:15:23] Grumpy, middle-aged Luddites.

DR: [00:15:23] Everyone that I know that’s-

Scott Bicheno: [00:15:24] Oh, me too, mate. Me too.

DR: [00:15:26] Everyone I know that’s been anti-AI, and then they start to kind of dabble.

Scott Bicheno: [00:15:30] Yeah. They start to see the light.

DR: [00:15:32] They don’t go back.

Iain Morris: [00:15:32] That’s why I don’t use it.

DR: [00:15:33] They love it.

Iain Morris: [00:15:36] Yeah. I don’t want to fall in love with it. I don’t want to be like that guy in the film.

Scott Bicheno: [00:15:37] He doesn’t want it to become his precious. Ah, my precious ChatGPT.

DR: [00:15:44] Like your girlfriend?

[00:15:44] “Write a salacious article about the public cloud. There was another outage, I’m so excited. Do it in the style of…”

Iain Morris: [00:15:49] No, that’s it.

DR: [00:15:52] No. Well, one time I wrote a prompt for you in one of my newsletters, which was like… I built a prompt for Iain Morris where it starts with some sort of historical reference to some calamity to start my article.

Scott Bicheno: [00:16:03] Oh, is this his opening paragraph?

DR: [00:16:06] Right? It’s his opening paragraph. because he has a very distinct style-

Scott Bicheno: [00:16:0] No, I often tease him about it on the pod.

DR: [00:16:11] Right? And I was like, “Write an article in the style of Iain Morris on this topic.” And it starts out with a little…

Iain Morris: [00:16:18] Yeah, I think Geoff Hollingworth did the same at some point. Yeah.

Scott Bicheno: [00:16:20] Yeah, he put it out on LinkedIn, didn’t he?

Iain Morris: [00:16:20] Yeah.

DR: [00:16:23] Yeah, and I put it in my newsletter.

Scott Bicheno: [00:16:23] Can I say a quick minor tangent? You were talking about maths, or math if you’re American. They only got one math over there.

DR: [00:16:29] Yes.

Iain Morris: [00:16:30] Telecom and telecoms, isn’t it?

Scott Bicheno: [00:16:32] Yeah, they got… Yes.

DR: [00:16:33] Telecom, yeah.

Scott Bicheno: [00:16:34] And there was a thing, how long ago was this? What’s the date on this story? Why don’t you put a date on your story? Where OpenAI claimed, some people from OpenAI claimed that OpenAI had solved these maths things that had been unsolved. Like these real Good Will Hunting-

DR: [00:16:49] Proofs, yeah.

Scott Bicheno: [00:16:50] Yeah, yeah, sort of things.

Iain Morris: [00:16:51] That only Matt Damon can do normally.

Scott Bicheno: [00:16:53] Exactly. I think they were called Erdős problems. I don’t know how you pronounce the accent above the O. And it turned out what it had done, it was very embarrassing for OpenAI, because it turned out what it had done, it hadn’t solved anything, it hadn’t done any maths. It had just done a really good search engine job of going around the internet and finding some other people who’d already suggested problems for this-

DR: [00:17:14] Solved it.

Iain Morris: [00:17:15] Right. But that’s the… Yeah, because that’s the shortcoming, I suppose, of AI at the moment. I’m not saying it will always be that way, and actually maybe we don’t want it to be as good as that, to come up with new solutions to things. Because that’s innovation, isn’t it?

Scott Bicheno: [00:17:26] On its own, yeah.

Iain Morris: [00:17:27] It’s not going over what’s already on the internet, and putting it into-

DR: [00:17:32] Yeah. I mean, we haven’t hit, what is it, artificial general intelligence where it’s actually….

Scott Bicheno: [00:17:35] That’s when we really start freaking out.

Iain Morris: [00:17:36] And maybe never will, but…

DR: [00:17:37] Probably. But you still need humans to guide it.

Scott Bicheno: [00:17:40] You think we’re paranoid now?

DR: [00:17:41] You guys especially.

Iain Morris: [00:17:44] I met Mavenir here this week. And I can say this, because it was on the record, the guy who’s their AI guy. And I was asking him what level… We might talk about this in a bit, but the levels that the TM Forum’s come up with for measuring how automated your network is, and zero basically means people are running around and pushing buttons.

DR: [00:18:00] Right, manual.

Iain Morris: [00:18:01] To where five is like…

DR: [00:18:01] Robot.

Iain Morris: [00:18:03] There’s like a CEO having a coffee and everything else just runs. And I said to him, “What is level five automation?” And I didn’t expect this answer, but he went on, “It’s basically like the Terminator.”

Scott Bicheno: [00:18:11] Right. And you’re like, “See? That’s what I keep telling you. Why don’t people listen to me?”

Pierre: [00:18:16] Wait, one more thing about AI that’s going to freak Iain out. I was at a friend’s house last weekend and he works for Google, and he has Gemini read his email, and then basically answer… And then he checks it before he sends it, but the AI kind of reads everything and goes…

Scott Bicheno: [00:18:31] Does the answer.

Pierre: [00:18:32] “This is where you should answer.” And then he…

Scott Bicheno: [00:18:34] Then he goes, “Yeah, go on, then.”

Pierre: [00:18:35] He tweaks it and then, yeah.

Scott Bicheno: [00:18:35] Right.

DR: [00:18:36] Perfect.

Pierre: [00:18:37] And he’s trained Gemini to be perfectly fitted for himself, but…

Scott Bicheno: [00:18:40] Well, I mean, that’s actually a good way to bring it back to…

Iain Morris: [00:18:44] What does he do with all the extra time that he’s got now?

Scott Bicheno: [00:18:45] That’s good point.

DR: [00:18:45] More work.

Pierre: [00:18:46] He’s just more productive, yeah.

DR: [00:18:49] Yeah, more work.

Scott Bicheno: [00:18:49] He gets pissed.

DR: [00:18:50] You can do so much. It’s amazing.

Scott Bicheno: [00:18:52] We should come up… The three of us, we’ll come up with a new AI product called the Pub Bot, where you can go down to the pub, but your boss… it’ll look like you’re working.

DR: [00:19:00] Oh, that’s great.

Iain Morris: [00:19:02] I’ve got a good way of getting it into telecoms.

Scott Bicheno: [00:19:03] Well, actually… Oh yeah. No, you go.

Iain Morris: [00:19:05] No, I was just going to say, because we’re talking about it in a kind of general sense at the moment. And generative AI… I mean, AI has been around a long time, but the generative AI is, what, 2022, I guess is when-

DR: [00:19:15] Mm-hmm. Like, November of ’22.

Iain Morris: [00:19:16] Yeah. So from your observations and your meetings with people and what you’re doing commercially, what difference has it made to telco so far? What are they actually doing with generative AI that they weren’t doing with… I mean, this thing that you were at in Germany, for instance, was a lot of that about how they’re using generative AI in their kind of networks, or…

Scott Bicheno: [00:19:35] Yeah, why don’t you give her a softball, mate? Only joking. I’m just winding you up.

Iain Morris: [00:19:38] Well that’s not a…[laughter]

DR: [00:19:41] Well this is a good question, because I think people have been using predictive AI, like machine learning, for a while. We’re not talking about that right now. That’s sort of like old AI or done and dusted, or everyone gets that. Everyone is talking about-

Iain Morris: [00:19:55] But still used and still useful.

DR: [00:19:56] Totally useful.

Iain Morris: [00:19:57] Yeah.

DR: [00:19:58] Great.

Scott Bicheno: [00:19:58] But it’s a bit so-last-decade in terms of what everyone’s talking about.

DR: [00:20:00] Well it’s just, it’s not the cool hotness. Generative AI is brand new, like you said, a couple years old. And I think when you go and talk to telcos, they don’t know how to apply it, they don’t have people inside using it. They’re still doing things kind of the old way, with pilots and RFPs. And that’s just not going to work.

[00:20:20] And what I talked about on my panel is, you really got to get this from zero to one. You don’t have any real kind of app production, widespread use cases, and you need to get from zero to one. And once you get to one people are like… It really is an eye-opener and an idea generator. And people are like, “Oh, we can use it for this and that,” whatever. And then off it goes. But there’s a lot of resistance from going from zero to one, right?

Scott Bicheno: [00:20:44] Well you’re in good company for that-

DR: [00:20:45] And then everyone’s talking about network applications, and I’m like, you’re never going to let your network do this because of all the fear that… Hallucinations or it’s not correct, and are we really going to let it go wild? So I advocate, start with somewhere safer, right? Less risky.

[00:21:02] And the easy place to start is customer support. It’s structured data with a ton of human-generated correct answers, because there’s solved tickets. Just go build a RAG database on your solved tickets.

Scott Bicheno: [00:21:14] I tell you, I’ve got a little tangent for that. My wife bought a bunch of stuff from Zara, and they sent completely the wrong stuff, just random stuff. And she’s having a f**king nightmare dealing with their customer support.

[00:21:26] And one of the reasons is, and this isn’t a knock on automation or AI in general, this is a knock on Zara’s specific implementation… If anything, this supports your point. They’ve implemented it so badly that she’s just getting absolutely nowhere. She’s being put in this bureaucratic loop where one thing sends her to another thing, which sends her back to the first thing. There’s no escalation to a human being at any stage.

[00:21:46] And this is Zara, a world-class brand, and they’re screwing it up such that I’m quite happy to slag them off on this pod and encourage my wife to not shop with them again.

DR: [00:21:54] Maybe they’ll help your wife… The return code is RMA.

Scott Bicheno: [00:21:58] Yeah. But I mean… But that’s just an anecdotal illustration of pretty much what you’re saying about… Especially if you’re going to deal with people like me, where I’m kind of on a skeptical hair trigger with this stuff, you better do it right. Don’t rush it in, it’s not a race, despite what the politicians would have us believe. Just do it properly.

DR: [00:22:16] I think, yeah, you scaffold it up over time, for sure.

Iain Morris: [00:22:19] How is generative AI helpful in that kind of area, that sort of customer area that you’re talking about? You feed in a lot of the information that you’re getting, you’re sort of doing a small language model, maybe, I guess equivalent? Or you-

DR: [00:22:30] Well, no, you use the big guys, right?

Iain Morris: [00:22:31] You use the big guys and you just tweak it, but…

DR: [00:22:32] Yeah, you just do model tuning. You do this thing where it’s called retrieval-augmented generation, where it’s using a database of knowledge to search and learn.

[00:22:42] And so in telco, where 60% of customer support tickets are a question about the bill, “Why is it this much? Why was I charged this fee?” And so it’s usually a task of looking at, okay, what was the bill? Looking at your customer record of what changed? And then you probably have this question answered 10,000 times in your old tickets. Put that together, right?

[00:23:06] So the way that we’ve implemented it at Totogi with our customer support, and also at Skyvera where we get a lot more volume of tickets, is we start with the easy, easy use cases. “How do I change my password?” Simple. And then we keep giving it more and more and more, harder, harder, harder. So it’s taking on a bigger and bigger percentage of tickets. But we start with, what’s the big easy ticket that you really don’t need a human to answer? It’s like…Yeah. You’re just like, “Hit the, “Forgot password,” I’ll send you another…” Right? There’s some stuff like that. But then you go the hard stuff, still humans. And we just kind of eat into the pile of work, with the goal of 100% of tickets are answered by AI.

[00:23:48] But I mean, we started in March of ’23, and after about maybe 15 months we were at maybe 70% with the goal of 100%.

Iain Morris: [00:23:58] This is internal. This isn’t a customer-facing thing.

DR: [00:24:01] It’s customer facing.

Iain Morris: [00:24:01] It’s customer facing?

DR: [00:24:01] Yeah, yeah.

Iain Morris: [00:24:01] Okay, it is.

DR: [00:24:03] But we started with the easy stuff to not ruin our brand or whatever.

Scott Bicheno: [00:24:06] Exactly. Dip your toe in the water sort of.

DR: [00:24:07] We had a button of, we told them this is AI, it’s training, it may make mistakes. We set that expectation. We had a button of like, I want to talk to a human. Anyone could push it at any time and escape to the human. But after a while, it answers tickets overnight, on weekends, on holidays, and the staffing of having 24/7 support is hard, right?

Iain Morris: [00:24:30] Yeah. I think you should say what Totogi is a bit, and also you mentioned Skyvera there as well and people might not-

DR: [00:24:36] People that don’t know all about me?

Scott Bicheno: [00:24:38] Before we do that, I’ve got your enduring job title at Totogi is acting CEO.

DR: [00:24:43] It is. Everyone wants to change it.

Scott Bicheno: [00:24:44] And you’ve been acting for a long time.

DR: [00:24:44] I’ve been acting for a-

Scott Bicheno: [00:24:45] Why don’t you just drop the acting?

DR: [00:24:48] I don’t know. I’m just, I don’t know.

Scott Bicheno: [00:24:51] So I’m going to guess that-

Pierre: [00:24:53] At least she’s not rotating.

Scott Bicheno: [00:24:55] You created the company, yeah?

DR: [00:24:56] I did.

Scott Bicheno: [00:24:57] Yeah. And I’m assuming that you put acting because you thought, “I’m the creator, sooner or later I’ll get a day-to-day CEO and I’ll just be the-”

DR: [00:25:04] That was the original idea was-

Scott Bicheno: [00:25:07] Yeah, be behind the scenes.

DR: [00:25:07] I’ll get it off and going. It’ll be an overnight success and then I’ll get someone and then you won’t have to explain like, “I’m not stepping down. It’s not big news. I’m just acting and we hired someone, whatever.” But you’re right, it has been going on.

Scott Bicheno: [00:25:19] So maybe we can get a scoop for this pod that you’re dropping the acting.

DR: [00:25:22] Yeah. Well, maybe.

Scott Bicheno: [00:25:23] You can announce it.

DR: [00:25:23] I can’t tell you how many times when we’re doing something big and they’re like, “Are you still acting?” My own team.

Scott Bicheno: [00:25:28] This isn’t the first time.

DR: [00:25:29] I’m like, “Yes, I’m still acting. I’m not going to change it.” One time, I forgot to write acting and I wrote CEO and the team was like, “Are you dropping the acting?” I’m like, “No. Thanks for reminding me. I’ll add it back.”

Scott Bicheno: [00:25:39] Acting CEO, I’ve got a picture of someone sort of stepping on and doing some kind of Shakespearean dialogue with a skull in there. I am, therefore, I’m the CEO.

DR: [00:25:49] Yeah, I don’t know. It’s dumb but-

Iain Morris: [00:25:50] So Totogi is quite young.

DR: [00:25:52] In telecom years, absolutely.

Iain Morris: [00:25:56] How did that all start off and what is it and why are you doing that?

DR: [00:25:56] Yeah. So I started out with the idea to rewrite the software of the telco industry. And in telco terms, I’m a baby. I’ve been around since 2017 when I became CEO of Redknee, which I renamed Optiva.

Scott Bicheno: [00:26:12] And they’ve just been bought by Qvantel, I noticed.

DR: [00:26:15] Yes. Yes. Saved really.

Scott Bicheno: [00:26:18] I think so. I think they had some challenges.

DR: [00:26:21] They had like $113 million in debt and about $40 million of revenue and about $8 million in the bank and unprofitable.

Scott Bicheno: [00:26:29] That sounds like the British government.

DR: [00:26:31] So when I came into what is now known as Optiva, I guess, someone named it Qvaptiva or something.

Scott Bicheno: [00:26:38] Right. Qvantel’s a strange name because it’s got a V.

Pierre: [00:26:40] Is it Qvantel?

Scott Bicheno: [00:26:41] It’s spelled with a V though.

DR: [00:26:42] I always say Q-Vantel, but I think James-

Iain Morris: [00:26:45] It’s like a NVIDIA thing.

Scott Bicheno: [00:26:46] Yeah.

DR: [00:26:47] Yeah. It’s like awkward consonants next to each other.

Scott Bicheno: [00:26:50] Too many consonants together.

DR: [00:26:50] When I came in, that’s when I noticed. I was like, “You guys are doing software like the ’90s.” I went in, in ’17. There was absolutely no use of the public cloud even for things… I’m not talking about network stuff. I’m talking about basic stuff.

Scott Bicheno: [00:27:02] Just back office whatever.

DR: [00:27:03] And I was like, “Uh, this seems kind of obvious.” And then I ended up leaving Optiva in 2020 and I was like, “Let’s just do public cloud. And this is going to work. It’s come to every single industry, including governments. It will eventually come to telco too.”

Scott Bicheno: [00:27:17] And that’s why we’ve always… The amount of free publicity you get out of us, pretty much every time public cloud comes up, we name check you.

DR: [00:27:23] Yeah, yeah. The cheerleader. So there’s two sides to what I do. Well, three. There’s TelcoDR, which is the thought leadership. I have a podcast. I have a blog. I’m constantly all over LinkedIn.

Scott Bicheno: [00:27:33] Iain’s been on your pod, hasn’t he?

DR: [00:27:34] He has. Maybe one day you’ll come too.

Scott Bicheno: [00:27:36] Right? Well, I’m still waiting for my invite.

DR: [00:27:38] Okay. We’ll do it. We’ll do it. Lindsay’s here. She’ll write it down.

Iain Morris: [00:27:42] Make it so.

DR: [00:27:43] So previous to Optiva, I was, I called it a turnaround CEO by software companies that were for sale for whatever reason and rehab them for high profitability. And sometimes there was really great tech that you could revive. Sometimes it was just-

Scott Bicheno: [00:28:00] So that’s one of your core competencies is turning these things around?

DR: [00:28:03] Yeah.

Scott Bicheno: [00:28:03] Okay.

DR: [00:28:04] I’m actually really good at making crappy companies highly profitable.

Scott Bicheno: [00:28:07] Well, that’s a really good skillset. That’s a really good skillset because there’s a lot of people who have the opposite skillset who get good companies and turn them to s**t. It’s nice to know that some people are doing it the right way.

Speaker 2: [00:28:17] And they still walk away with the money.

Iain Morris: [00:28:18] We know a few of those, don’t we, Scott?

Scott Bicheno: [00:28:19] No, we don’t know any of them, Iain. Shut up.

DR: [00:28:21] So Skyvera does that. And what’s nice is it generates a very healthy profit that actually helps fund Totogi. And then Totogi is a Silicon Valley startup that’s actually in Texas. So it’s brand new software, totally living the dream.

Iain Morris: [00:28:39] So you’re Austin based?

DR: [00:28:40] Yeah, I’m in Austin.

Scott Bicheno: [00:28:41] That’s where Yeti comes from, isn’t it?

DR: [00:28:43] And Yeti is down the street from my house.

Scott Bicheno: [00:28:44] So you’re very brand loyal to Yeti.

DR: [00:28:46] I know.

Iain Morris: [00:28:46] And Matthew McConaughey.

DR: [00:28:48] Yes. All right, all right, all right.

Scott Bicheno: [00:28:48] And Joe Rogan.

Iain Morris: [00:28:50] There you are. In fact, I saw Joe Rogan in Austin.

Scott Bicheno: [00:28:52] Oh yeah, you went to the mothership, didn’t you?

Iain Morris: [00:28:54] That picture was drawn in Austin that’s on the wall there, which is a picture of us. Although-

Speaker 2: [00:29:02] Iain was not pleased to have to wait outside. I was like, “Trust me. Trust me.”

DR: [00:29:06] Yeah, yeah, yeah.

Scott Bicheno: [00:29:07] You made it happen, didn’t you?

Speaker 2: [00:29:08] I bribed the doorman.

Scott Bicheno: [00:29:09] You slipped the geezer a tenner or something.

DR: [00:29:10] So it’s a little bit of a symbiotic relationship where if I buy some tech that could help Totogi, Totogi will use it. Totogi is building new products that maybe Skyvera customers are interested in. And so the strongest example is I bought CloudSense, which is a telco focused CPQ, configure price quote product, sits on top of Salesforce with a fabulous list of tier one customers. And we’ve been able to go to nearly half of them and talk to them and get them to use BSS Magic. And when they see it working-

Iain Morris: [00:29:44] It’s like Skyvera is like the fund. It’s like a fund?

DR: [00:29:48] A little bit, yeah. It’s the profit. So we use that profit… I’m still returning a profit, but I use that money if I want to. I use it to either buy more companies or fund Totogi.

Iain Morris: [00:30:02] And Totogi’s in one specific part of the BSS/OSS space, which … I’m grouping it as BSS/OSS because traditionally, that’s how people talked about it. I know things may be a bit different now, but it’s in one particular-

DR: [00:30:10] Yeah. It’s focused in BSS because I think when it comes to AI, it’s going to be the tip of the spear versus network. I think it’s much harder to break in as a startup, small company with new software on the network side.

Iain Morris: [00:30:23] So where would Totogi come in where people have been using traditional solutions, let’s say, pre-2017 stuff. And who are you going up against, I guess? Who were the traditional guys?

Scott Bicheno: [00:30:36] What’d you bring to the table apart from beer?

DR: [00:30:37] We started with a charging product because that’s what I knew from Optiva. So we have a charging-as-a-service product that runs completely in the public cloud.

Scott Bicheno: [00:30:44] Is charging the same as billing?

DR: [00:30:46] No, it’s rating. So it’s more real time than billing. Billing is more invoicing.

Scott Bicheno: [00:30:52] Oh, okay. Right, right.

DR: [00:30:52] Like in a prepaid situation.

Scott Bicheno: [00:30:54] So it’s a more dynamic real time type of thing.

DR: [00:30:56] Right. It’s like for prepaid, I’m going to go check your balances before I let you download that YouTube video. Oh, you’re out of balance, you need to top up.

Scott Bicheno: [00:31:03] I hear you now.

DR: [00:31:04] So it’s like an ATM automatic teller machine. It’s more like a money machine or it’s a monetization engine for-

Scott Bicheno: [00:31:13] Yeah. So it’s real time to use your ATM metaphor, so you can’t withdraw some money from one and then-

DR: [00:31:15] I can’t pull down. Right, and the telcos are like, “I’m not going to give you like one penny extra of data.”

Scott Bicheno: [00:31:21] Yeah. Especially not on prepaid.

DR: [00:31:22] Exactly. Because they don’t know if they’re ever going to top up again and they’re just giving way free sales.

Iain Morris: [00:31:26] Is that what you used to do when you were at university?

Scott Bicheno: [00:31:29] What, try and rip off the ATM?

Iain Morris: [00:31:30] Go to one ATM and then run to another one before-

Scott Bicheno: [00:31:32] I certainly remember-

DR: [00:31:33] Race condition, literally.

Scott Bicheno: [00:31:35] I certainly remember testing my overdraft limit a few times, but even a drunken 18-year-old Scott could be outsmarted by their system.

DR: [00:31:43] When I was at university, I specifically picked a bank that would allow you to withdraw $5 at a time.

Iain Morris: [00:31:50] Oh, to limit yourself?

Scott Bicheno: [00:31:51] You were more sensible than I was.

DR: [00:31:53] So the banks have a minimum withdrawal number and it was like 10 or 20. I needed five.

Scott Bicheno: [00:31:59] Because that’s all the money you had.

DR: [00:32:03] I had like seven and I couldn’t access it. I’m like, “I got to go to the bank that would let me pull out five so I could have $2 left over.” Poor. It was brutal.

Scott Bicheno: [00:32:11] Yes.

Iain Morris: [00:32:11] So the guys who’ve been in telco software for years would be doing charging solutions, I guess. What’s wrong with that stuff?

DR: [00:32:18] Yes. So we go up against Ericsson, Amdocs has their open net purchase. Nokia had a charger, sold the chargers, building a charger again. Optiva has a charger and so do we.

Iain Morris: [00:32:28] So it’s a pretty competitive market already?

DR: [00:32:30] It really is. And it’s a heart transplant because it’s your monetization engine. And so our claim to fame there was Zain Sudan had a … It’s one of the most active civil wars on the planet and the rebels attacked on premise. They had Ericsson BSS and charging running in two separate data centers in the capital of Khartoum and they got bombed. Electricity got cut and they were just dead and they did it also to MTN. So there was like just zero mobile service. This was right before MWC24. And we had been talking to them about a backup to their backup using cloud and would it work? And we designed it on paper, but they were sort of stalling. Well, about two weeks before MWC24, they called us and they said, “We’ve just been cut. We’re dead. We have no access to anything. Can you get charging up in two weeks?” And I’m like, “Guys, you would give anyone in the world 18 months to stand up a charger and you’re giving me two weeks? PS, I think I can do it. ” And it took me 18 days. That’s so awesome.

Scott Bicheno: [00:33:38] I just teased Iain about giving a softball question, but I’m going to give you the ultimate softball question. Danielle, have you recently demonstrated spinning up BSS from nothing in a very short amount of time?

DR: [00:33:49] I did. I did. so that’s the other big product that we’ve been working on, which is BSS Magic. And I talked about this also at MWC24 with the vision of empowering CSPs and operators to build their own BSS with AI. And I think that is their dream, right? Not speaking from a vendor perspective, but if you go talk to like a Lester Thomas or a head of IT, they’re like, “We just want to write it. ” But it’s been just hard to maintain. It’s wide. It’s huge. There’s no one definition of BSS. You could say it’s five modules, you could say it’s 25 modules, right? It’s different things to different people. They’re using it in B2B, B2C situations, IoT situations. Oh, we’re going to stand up a down brand. We don’t want it to touch our main brand. So we have another BSS for that.

[00:34:33] So it’s a very complicated situation and I stood on stage and said, “That’s what we’re working towards, allowing you to write your own BSS.” And I remember looking in the audience, people shaking their heads and scoffing, like literally scoffing at me and I’m like, “I’m going to do…” I felt like Elon Musk.

Iain Morris: [00:34:49] So it’s a tool to build a BSS?

DR: [00:34:50] Yeah.

Iain Morris: [00:34:52] Okay.

DR: [00:34:52] Right? And that’s not like it’s only-

Iain Morris: [00:34:54] Well, what happens to your charging product then if they can just do their own or is it-

DR: [00:34:59] They can use it if they want or not or whatever. Yeah. Just it’s there. It’s not going away.

Scott Bicheno: [00:35:01] So you’re not worried, which I think what Iain was asking, you’re not worried about cannibalizing your own stuff?

DR: [00:35:08] Nah.

Scott Bicheno: [00:35:09] That’s actually the most forward way of looking at things. In fact, on the last pod, we invoked that venerable example of Kodak that sat on digital photography technology for ages because it had such a cash cow in film and then that all went horribly wrong. So you can’t-

DR: [00:35:22] Yeah, you have to.

Scott Bicheno: [00:35:23] You can’t be too afraid of cannibalizing your own cash cow.

Iain Morris: [00:35:25] But on these BSS systems that are in place at the moment are fairly complex things with, I’m guessing… they are software based, aren’t they? It’s not like we’re talking about boxes and optical equipment and this sort of thing, but they’re very complicated software tools that link to all parts of the business. I’m not in this area obviously, so I’m trying to understand the nitty-gritty, but the idea of somebody being able to come along and replicate that from AI from scratch sounds-

Scott Bicheno: [00:35:51] In a day.

Iain Morris: [00:35:51] People may say it sounds far-fetched, I guess.

DR: [00:35:53] And insane. It does. And it works, which is so crazy. Our idea here is not… A tier one CSP isn’t going to say, “Hey, let’s replicate our entire BSS stack with this Totogi, write our own thing and deploy it overnight.” We don’t think that’s how it’s actually going to go down. The demonstration over the last couple of days in Dusseldorf was more about if you can do this, you can do everything easier than this. This is really hard. And so we’re not even arguing this is like-

Iain Morris: [00:36:25] It’s like filling gaps at first probably with how it could be used?

DR: [00:36:28] Well, Lester Thomas, again, I love this quote. He was on a podcast with TM Forum in, I think it was 2021.

Iain Morris: [00:36:34] He used to be at Vodafone, yeah?

DR: [00:36:35] He is still at Vodafone.

Iain Morris: [00:36:36] He’s still at Vodafone? Okay.

DR: [00:36:37] Yeah, he’s at Vodafone Group. He’s one of the main guys that came about with TM Forum’s ODA, their open architecture that tries to solve this vendor interoperability problem, which is he has 5,000 BSS products installed across 20 some odd OpCos, 5,000. They’re not all from the same vendor, right? They’re homegrown systems, mainframe systems, legacy systems, Amdocs systems.

Iain Morris: [00:37:04] This is just a product of consolidation that’s gone on over the years. You’ve picked up new businesses, you’ve bought new things and bolted it on.

DR: [00:37:10] Well, like I said.

Iain Morris: [00:37:10] Before you know it, it’s just one big-

DR: [00:37:13] You have a B2C stack, which is your main stuff. You have a B2B stack, which you’re like, “Well, that has different needs.” We just came up with a down brand of whatever. We bought something. And so over the years, you’ve stitched them together. It’s a pile of spaghetti mess. So when you want to replace… Let’s say one vendor makes you mad or goes out of business or whatever, right? The Optiva thing happens and you’re installed there. You’re deeply in the middle of the spaghetti pile. How do you take that one noodle out without screwing up everything? And they’re stuck, right? And it’s one of the reasons why they don’t have agility and they can’t move fast.

[00:37:47] Launching an iPhone 17, it’s probably a six-month process to get the website updated, the catalog ready, right? The plans ready, network ready, right? Like, “Oh, we’re going to support eSIM on this launch.” It is not something like, “Oh, new phone came out, no big deal. Let’s go live in one day.” It’s a pain. And so they talk about moving at the speed of tech codes. Why can’t we be that good? And I’m like, “It’s this legacy crap. We got to get out of it.”

[00:38:14] So BSS Magic attempts to solve that problem and not by saying, move to a 100% Totogi system. I’m actually saying keep exactly what you have installed. We call that layer the data layer, right? We connect to that. It doesn’t matter what system it is. And then next-

Iain Morris: [00:38:31] This is through these APIs that you’re talking about, right?

DR: [00:38:33] It could be APIs, it could be MCP, model context protocol from the AI vendors. There’s lots of different ways to access it without-

Iain Morris: [00:38:41] So you don’t need ODA to do it?

DR: [00:38:43] You don’t need ODA to do it.

Iain Morris: [00:38:45] Okay. Almost makes you wonder why ODA is necessary if you can use MCP or something.

DR: [00:38:48] It’s useful. It’s a nice standard that everyone’s agreed on, right? And a lot of vendors-

Iain Morris: [00:38:53] TM Forum will be shooting me when I leave the pod but-

DR: [00:38:56] There’s a lot of vendors.

Scott Bicheno: [00:38:57] Exactly. That why I was how daring you about.

DR: [00:38:58] There’s a lot of vendors that support it. So that’s nice. Even Amdocs supports it. They’re moving right up the API certification leaderboard that TM Forum maintains and reports on, which is great. And so okay, now I can connect to it. And then our middle layer, which is I think where our IP really sits, is this ontology layer that understands telco. It is the context engineering. It expects to talk to BSS. It knows about BSS. We call it the vocabulary of telco, the nouns and the verbs.

Iain Morris: [00:39:28] Sorry, go on.

Scott Bicheno: [00:39:29] That word ontology.

Iain Morris: [00:39:30] Yeah, it’s come up a few times recently.

Scott Bicheno: [00:39:33] Yeah. I actually linked it. So I wrote up your story, what you just achieved in Germany. Simple headline said, Totogi automates BSS code writing. And your claim prior to it, while I’m being generous, I might as well tell you what your claim was.

DR: [00:39:44] I love it. I think I should just bring you more beers. That’s really why I do it.

Scott Bicheno: [00:39:48] I’m such a cheap date.

DR: [00:39:51] Beer and koozies.

Scott Bicheno: [00:39:52] You said that you were going to build 500,000 lines of BSS code in one day. And then Jen, who’s patiently waiting outside from Liberty, sent me and Iain a followup email today to say what you actually did achieve. So you’re going to aim for five core modules plus an audience rated feature. So I guess that’s your way of saying this isn’t pre-prepared. You come up with something-

Iain Morris: [00:40:14] So this is what you’ve just done.

Scott Bicheno: [00:40:15] This is what they’ve just done. Yeah, yeah.

DR: [00:40:16] Yeah. We really started with the premise of no one believes us. The CSPs are skeptical and I think there is a lot of AI slop out there. Slideware saying they have it.

Scott Bicheno: [00:40:26] It’s attached to everything right now, isn’t it?

DR: [00:40:26] How do you prove it works? And we’re like, “We’re going to bring a coder and code in front of you.” If you guys have a better idea of how to prove it, tell me.

Scott Bicheno: [00:40:38 ]No, I think as I wrote in the conclusion to the piece I wrote about yours, I said… I’ll just read myself back. “While we have no reason to question Totogi’s claims, it’s hard to avoid wondering if there’s a catch to all this AI expedited coding. So many utopian claims are being made on behalf of AI these days that sometimes turn out to be too good to be true.” And I linked to that thing about the maths that I mentioned earlier. And then just one more sentence, sorry, mate. “But it’s not for us to assess the merits of this offering, only to tell you about it. Totogi will doubtless be delighted to do another live demo for any potential customer that expresses even passing curiosity.” Sorry, mate. I’m shouting over-

Iain Morris: [00:41:10] No, I think the live demos is great because obviously that’s the way to shut up-

DR: [00:41:14] Yeah. Anyone who asked us, I think we made over 300 commits to production. So that’s how many times we shipped to a place where AI thought it was ready to go and anyone can go play with it. We let anyone who wanted to touch it at the show, play with it.

Iain Morris: [00:41:29] So the ontology thing you’re talking about though, how is this done basically? Was this a case of leaning? You were talking earlier on about these large language models that have come out of OpenAI and Anthropic was the other one I think you mentioned. Was it based on working with one of you?

DR: [00:41:44] So it’s definitely LLM, right? There’s a LLM involved.

Iain Morris: [00:41:48] You’re going to tell us which one though.

DR: [00:41:51] Well, I think we’re using Claude Code for the moment because it’s the best coder.

Scott Bicheno: [00:41:55 ] That would’ve been my assumption on that basis.

DR: [00:41:58] We like it a lot, but it’s set up in a way that if something better came along, we could switch. It’s set up in a way that if a CSP like SKT is really deep with Anthropic, I think T-Mobile spend $100 million with OpenAI. We’re set up in a way where if a CSP is like, “I have a model preference,” we could switch.

Scott Bicheno: [00:42:15] And there are lots of good reasons why you don’t want to be monogamous anyway.

Iain Morris: [00:42:18] And you’ve thrown lots of data at that, that you’ve got from your experience.

DR: [00:42:22] Right. We’ve added a lot of telco context, right? And so like I said, it expects to be interacting in BSS, in telco. That’s what it’s always expecting.

Iain Morris: [00:42:36] Because if it’s going to be that good, then it’s going to be capable of working with all these different systems and vendors.

DR: [00:42:42] That’s what makes it so good. I think is that piece. I do use Cursor, which is a horizontal coding development environment. Very, very popular, probably the best one right now on the market that a lot of software engineers are using. We use Cursor because we still need a coding environment. We use these other tools, LangChain to create agents. We use these horizontal tools, but the key is that context engineering. It’s going to create something that you’re really what you want for this scenario for telcos. You can’t take this and say, “Okay, let’s go to a manufacturing setup and go…” I could not generate 500,000 lines of code for BMW, right?

Iain Morris: [00:43:17] Yeah, yeah. So this-

DR: [00:43:18] Because I’ve done a lot of work on the ontology.

Iain Morris: [00:43:21] So this has been worked on for like a couple of years?

DR: [00:43:23] Two years.

Iain Morris: [00:43:24] Two years? Okay.

DR: [00:43:24] Yeah. And I’ve gone through a lot of dev teams, right? I’m like, “You guys don’t get it. You’re not getting anywhere.” And then we finally had a big breakthrough, I would say in the last nine months where finally it was like, okay, it’s exactly what we’re looking at. But true story, right? Deutsche Telekom, obviously in Dusseldorf, they have a big office there, they’re like, “Hey, can it do this? ” And we’re like, “Go to your panel and on the next break, it’ll be done.” And it was.

Iain Morris: [00:43:54] Right.

DR: [00:43:55] It was a CR.

Scott Bicheno: [00:43:56] Well, if you can actually, because that was the point of the final paragraph of my story about this is it’s all very well you’re demonstrating it in what’s still, if you were skeptical, which some people are.

DR: [00:44:08] Yeah, they will be.

Scott Bicheno: [00:44:09] If you’re skeptical, could think is maybe a bit of a curated environment or something like that. Whereas if you’re doing it In Vivo in front of people, I would have thought that’s quite compelling.

DR: [00:44:18] Yeah.

Iain Morris: [00:44:19] What’s the job of the guy at the network?

DR: [00:44:22] The human? The one guy?

Iain Morris: [00:44:22] Yeah. So what does he do? He goes, “Oh, I need a BSS thing to fill in the gaps here and I’ve got this problem to solve.” But what’s left for him to do then? Or her.

Scott Bicheno: [00:44:34] Or her.

Iain Morris: [00:44:34] Sorry.

DR: [00:44:34] Yeah, no, they’re great curious questions. So he is scheduling the work a little bit, watching it work. Sometimes it comes back.

Scott Bicheno: [00:44:43] It is a him, you’re lucky.

DR: [00:44:44] It’s stuck or something and needs more help. Again, he is the master brain. He knows what we’re about to go build. He knows all the features and how they connect, what the architecture should be. He knows.

Iain Morris: [00:44:54] But he’s not actually coding or she’s not actually coding. They’re putting in prompts and this sort of thing?

DR: [00:44:58] Correct.

Iain Morris: [00:44:59] They are? Okay.

DR: [00:44:59] Zero writing of one line of code or looking at the code. This manual-

Iain Morris: [00:45:06] So sophisticated prompts is basically what this person’s doing?

DR: [00:45:10] Right. There’s prompts built in because again, it knows that we’re going to build a BSS today. It knows that context. So he doesn’t have to work-

Iain Morris: [00:45:16] It’s not like I could do this, is it?

Scott Bicheno: [00:45:18] Yeah, I was going to say, following on from that, what skillset does that one individual that you use for this?

DR: [00:45:22] Telco and AI is really what you want.

Scott Bicheno: [00:45:25] So they need to be good at both?

DR: [00:45:26] Correct.

Scott Bicheno: [00:45:27] I was going to follow up, I went on a tangent, not like me, is it? Earlier when we were talking about ontology, because I heard we went to… I linked to this in the story about your thing as well. We went to an Ericsson BSS/OSS thing and there’s a chap from Telstra, I think, and he was going on about ontology and it was the first time I’d heard it. For you, this would be old news, but it’s the first time I’d heard it and I looked it up and I noticed at the time, it’s got a broader philosophical, like the study of the meaning of life sort of thing. But in the context of AI, at least the way he described it, and I’ll pass it to you to refine this description, it’s about making the data and the AI more useful. And part of this is through packaging the input in such a way as to optimize the output. That was one of the takeaways I got from it. But you tell me, what do you mean by ontology in this context?

DR: [00:46:21] We mean the nouns and verbs of the industry. So example nouns would be subscriber, device, SIM, cell tower.

Scott Bicheno: [00:46:31] I see.

DR: [00:46:31] Right? And then the verbs are like-

Scott Bicheno: [00:46:33] And understanding the significance of those.

DR: [00:46:34] And then the verbs are like what you can do, like onboarding, deactivating, provisioning. And so when you put the nouns and verbs together, you make sentences. How do they interact with each other? How do they communicate with each other? What are the different states they might have? What are the different business logic you might want to add to them like the concept of types of subscribers, premium versus prepaid versus postpaid, VIP versus high ARPU VIP versus regular guy. And so these are all the things that telcos, when you think about the business of a telco, the language that goes on inside there.

Scott Bicheno: [00:47:14] Is quite specific.

DR: [00:47:14] That’s what we’re trying to encapsulate.

Iain Morris: [00:47:16] Yeah.

Scott Bicheno: [00:47:16] Makes sense.

DR: [00:47:17] To train AI. And so I did this five minute speech right in front of a panel, really, really short, but this thing knows telco like your best experts know telco. Imagine if you could channel Santiago Tenorio, Yago, who was at Vodafone and now at Verizon. He’s maybe one of the best thinkers of network currently in our industry. What if you could channel his brain to think about your network design?

Iain Morris: [00:47:44] Yeah. He wouldn’t like that.

DR: [00:47:46] He wouldn’t like that.

Iain Morris: [00:47:46] He probably would. I don’t know. You’d have to ask him.

Scott Bicheno: [00:47:46] Okay.

DR: [00:47:46] But your best-

Scott Bicheno: [00:47:46] Yago bot.

DR: [00:47:52] Right. But your best experts, right? How do people interact just in any business?

Iain Morris: [00:47:57] But is one challenge, generative AI makes things arguably easier for people. If you want to write an essay, you can put in a simple prompt and get it done and you don’t have to put in the hard work. But it’s one challenge with BSS Magic that it still requires someone who’s very, very skilled to be able to take advantage of this. So you not only going to have the telco knowledge, but the AI knowledge as well and be able to combine those. It’s not like you could put Scott in there and get him to do this stuff. Sorry. Don’t know why I use Scott as an example.

Scott Bicheno: [00:48:26] An example of a complete dunce-

Iain Morris: [00:48:28] But is there oddly a skills gap in terms of getting this sort of thing out? Because the telcos, they might have the domain knowledge, but they don’t have the AI knowledge.

DR: [00:48:37] Well, so they don’t have people that we talked about at the very beginning of the pod here. They don’t have the skills of people who know how to wield these AI development tools. And that’s been super key for us is building up a team that knows that and then coupling it with telco knowledge. But we’ve encapsulated a lot of the telco knowledge in there and so kind of easing the load on our, we call them forward deployed engineers easing the load on those guys where it’s more of like, okay, really focus on the AI because a lot of it is built in, but just make sure architect the system so that you’re just watching it, you’re babysitting the code, you’re making sure it’s getting done, and then sometimes it’ll say like, “Hey, I need help.” And then the architect goes in and it’s like, “Okay.”

Scott Bicheno: [00:49:22] So it does do that.

Iain Morris: [00:49:25] If you get a rush of telcos wanting to take advantage of this, as I’m sure you would be very happy to have, would they need you guys to go in and train them up and show them how? Yeah, there’s a service side of it

DR: [00:49:35] Yeah. Actually, based on the success of the event this week, thinking about launching a service to train people for telcos at scale. So I’ll give you an example. Something that’s happening in Austin is a cool program called Gauntlet AI. My son actually went through it this summer and it’s a 10-week program, three weeks remote and seven weeks in Austin to teach you how to code with AI, and the monetization, it’s completely free for the participant. You’re guaranteed a $200,000 job offer at the end. And the way that the owners, I know it’s nice, the way the owners monetize it is at the end of the program, they’re letting companies hire these individuals. So they come in not really trained on AI and they leave very skilled on AI. And so it’s a nice-

Scott Bicheno: [00:50:22] And the air industry is chucking money at talent.

DR: [00:50:24] I mean there was like $100 million starting bonuses that people turned down. Yes.

Scott Bicheno: [00:50:31] Iain, do you remember we were in the-

Iain Morris: [00:50:33] We’re in the wrong job, Scott.

Scott Bicheno: [00:50:35] We went to New York a little while back and one time we were out meeting his wife, Magda, and we were just sat in a beer garden and we heard these youngish women, let’s say I’m going to guess 30s or 40s young women anyway, and they were having a chat and one of them was going, “Yeah, they offered me a million to stay in. And I just said, ‘No way.’” And we were just like, f**king hell. Do you remember that right?

DR: [00:51:00] No, I think the $100 million Meta offer was turned down by the Open. So it is a mega war. And I think telcos, how are telcos going to develop this talent or acquire it?

Iain Morris: [00:51:12] Yeah, I wonder if they can afford to.

DR: [00:51:13] They can’t. And so they’re going to have to build it on their own. And I’m like, maybe because I’ve been building it. I helped the guy that runs Gauntlet AI, I helped him. I’ve run similar training programs and so I kind of advised him and saw the program. My son went through Cohort 2, he got six $200,000 job offers as a first year university kid. He’s like, “Mom, should I drop out?” And I’m like, “Absolutely not.” He’s like-

Scott Bicheno: [00:51:39] Why? 200 grand, Danielle.

DR: [00:51:41] Well this is what I said. I was like-

Iain Morris: [00:51:43] You just finish it later.

DR: [00:51:44] Well, I said that you’ve now established a floor for your market-

Scott Bicheno: [00:51:49] So it’s upward from there.

DR: [00:51:49]… and when you graduate, you’re going for a million dollars out of school.

Scott Bicheno: [00:51:51] f**king hell, Beers are on him.

Iain Morris: [00:51:53] He bought them.

Scott Bicheno: [00:51:54] No, but I mean good for him and good for you for obviously helping him get to that spot.

DR: [00:51:59] But he didn’t believe, right? He was… Recently divorced. I don’t know if you guys knew that. So I got divorced about two years ago-

Scott Bicheno: [00:52:06] For a minute. I thought you were saying your son was, I think he’s really not wasting any time, is he?

DR: [00:52:12] But I was married to-

Scott Bicheno: [00:52:13] No, I know you were because you were Royston and now you’re Rios.

DR: [00:52:16] I was Royston and now I’m back to my maiden name. But my ex-husband actually is a very brilliant software developer architect, right. Principal architect at Shopify. He’s now taken a CTO role.

Scott Bicheno: [00:52:26] So you’re saying your son’s genetically cheating?

DR: [00:52:28] Well, we both have computer science degrees. Right, but Benjamin doesn’t know how to code, right? He’s a freshman at Rice studying computer science, and we were talking about Benjamin going through Gauntlet AI before the program started and my husband is very skeptical of AI being able to generate code. And I’m on the very opposite of the spectrum of like, “It’s amazing. I’m building a product that’s writing code right now.” And Benjamin’s like, “Who do I listen to, my brilliant dad or…” I hope he says his-

Iain Morris: [00:52:56] Well, your husband’s point’s interesting. I know –

Scott Bicheno: [00:53:02] Ex husband. You just told you got divorced and you called him your husband. It’s confusing, Danielle.

DR: [00:53:03] It’s recent.

Iain Morris: [00:53:05] But I guess AI coders would have the same maybe resistance to this that a writer has to having AI produce something. And I know some people are concerned that I was chatting about this with someone earlier about AI doing all of the code and how you’re so far removed then from what’s actually going on that it becomes, actually there’s a guy on LinkedIn, I’ve got it here. And this wasn’t specifically to do with your thing but he-

DR: [00:53:29] Yeah, just a generic.

Iain Morris: [00:53:30] Let me look it up.

DR: [00:53:31] It wasn’t the troll last night on Twitter that was coming after me about this.

Scott Bicheno: [00:53:34] Oh, were you?

DR: [00:53:35] A little guy, yes.

Scott Bicheno: [00:53:36] He’s hassling you about this BSS Magic thing?

DR: [00:53:36] Yeah, he’s like-

Scott Bicheno: [00:53:39] He’s calling bulls**t on it, wasn’t he?

DR: [00:53:40] Yeah, he was.

Iain Morris: [00:53:41] He’s called Josh Anderson. I mean he’s one of those people. He’s got loads and loads of followers.

DR: [00:53:45] We can link his link in the show notes.

Iain Morris: [00:53:48] But he said, “I spent three months using AI to write every line of code for a new product, not because I couldn’t code, I’ve been doing it for 25 years. I needed to know what happens when you go 100% AI because every client asks me the same question, how should our engineering team use AI. The product shipped, it worked. I felt productive for about three months, then I needed to make a simple change and I realized I couldn’t confidently modify my own product. 25 years of experience, and I’ve become helpless looking at code I directed an AI to write,” and he goes on. But it’s emphasizing that point, I guess, that you become too far removed from what you are doing that it then becomes hard to step back into it.

DR: [00:54:23] And I was a developer, I have a computer science degree, so I totally get this world. And so when you’re a developer, you literally have every line of code pretty much in your head. When you really know a product, you know something’s wrong. You’re like, “Well, it’s in this module, it’s probably this thing. It’s probably this type of problem.” And you can go right to it and fix it and feel very confident about knowing where it is. And with AI, you just don’t need to do that anymore. I think it’s still important to architect it. And we’ve done that for the thing that we did over the last couple of days in Germany. We’re like, “Okay, this is what you’re building. It needs to be able to support this many subscribers. In these parts, it needs to be very performant.” So can’t just write anything. Just like in school, you learn what algorithms are best for things. And so you still need to know those things.

Scott Bicheno: [00:55:13] I don’t know what school you went to? We didn’t cover that in my school.

Iain Morris: [00:55:17] But I guess the concern is the stepping stones that you would start off on are not there. And I’ll use an analogy that you’ll really like. You are into tennis, but when they completely replaced line calls in Wimbledon this year, it’s happened in other tournaments, but Wimbledon, it was a big news story because they’d basically used Hawkeye to get rid of the line judges and me and I think a lot of people looked at that and thought, “Well, you can’t really tell if the balls hit the chalk because it’s too quick these days. It’s actually quite a good move.” But a lot of the umpires didn’t like it because they said that’s the training route for you to go through to then be an umpire. You spend time on the court, you learn how it all works, you get used to how things happen and then you go and become an umpire. If you get rid of those people, then you lose the kind of training route to… And it just seems to me like that’s a simple way of putting it. This is much more complicated and therefore potentially more risky.

DR: [00:56:08] I think you still need to have a computer science degree. I mean Benjamin through Gauntlet AI, he’ll hate this story, but I’m going to put it on the record. So the very first day, his assignment was to rewrite Grammarly, which is this little tool that writers use to help you fix your-

Scott Bicheno: [00:56:25] Some writers. Thank you very much.

Iain Morris: [00:56:28] You know grammar inside out.

Scott Bicheno: [00:56:30] I don’t need some goddamn computer to tell me how to write. Anyway, carry on.

DR: [00:56:35] You know MVP is due in 24 hours and Benjamin is not a coder. I mean he’s only had a couple of maths classes.

Scott Bicheno: [00:56:42] Thank you.

DR: [00:56:43] And maybe an intro to computer science. And he’s dabbled, but he has tears in his eyes. He is like, “Mom, I can’t do this.” I’m like, “Benjamin-”

Scott Bicheno: [00:56:52] It just seems insurmountable.

DR: [00:56:54] Because he is thinking like a coder. He’s like, “I can’t generate all the code that is going to be required to build this application.” And I’m like, “Benjamin, this is an AI training program, so use AI.” And I actually spun around at my desk and I did it right there. And I was like-

Scott Bicheno: [00:57:11] Using the AI? Just to show him how straightforward it is.

DR: [00:57:14] Five minutes. It took me five minutes. And I was like, “There’s a Grammarly clone.” Now I’m sure you have some other requirements to do. This isn’t the whole thing. I don’t know the assignment. And he looked at me and he was like, “I’ll be back, mom, I’ll be back.” And he worked all night.

Scott Bicheno: [00:57:27] That’s an appropriate catchphrase when we’re talking about AI, ” I’ll be back.”

DR: [00:57:31] I’ll be back. So he finished it and after that first week of Gauntlet AI, he was like, “Mom, you’re right and dad’s wrong. AI is going to be-”

Scott Bicheno: [00:57:42] I can see why you like that anecdote.

DR: [00:57:45] That’s great. So yeah. It’s just a different world. It’s a different way of thinking.

Scott Bicheno: [00:57:52] It is, it is. And I get that for all my naysaying and Luddite-ism, I get that it’s inevitable and that it can be very useful and it can be very productivity enhancing and all that sort of thing. And so much of it, we’re still such early days. I mean we’re only really talking about it in the way we are now since ChatGPT first came along a couple of years ago. And so much of, hopefully a good education is going to be, I don’t know specifically prompt engineering, but how to best interact with this incredibly powerful resource.

DR: [00:58:27] Absolutely.

Scott Bicheno: [00:58:27] Not only to make it useful for you, but also in a sort of safe way. And obviously as I’m sure you’re acutely aware, there’s industry springing up to do with ethics and safety and guardrails and all that sort of thing.

DR: [00:58:38] But I’ll say one thing as we probably are going to change topics here in a second-

Scott Bicheno: [00:58:41] Well, I got one last question, but you say that and then I’ve got one last thing.

DR: [00:58:46] Which is back in the day, we coded with punch cards, which required you to know 0s and 1s to make the machine do what you wanted it to. And I’m sure there was resistance to adopting higher order programming languages that was programming in English. C, the C language was if-then, right? And it would compile it down to the 0s and 1s.

Iain Morris: [00:59:06] Yeah, I’ve done that.

DR: [00:59:08] And so there was that shift. I think this is that same shift. It’s just going up higher on the chain. You still absolutely need to understand fundamental computer science principles and architecting and patterns and all that stuff, but I don’t think you need to know every single line of code to support a product anymore.

Scott Bicheno: [00:59:26] So my last question is kind of builds on that and it also is a follow-up on the question Iain asked about this concern about underlying human competence. So one of your answers, sorry I can’t put it into context, but one of your answers to one of Iain’s queries was like, “Well, you don’t need that anymore,” or something to that effect. My concern that I share with Iain to the point of this guy that he read out from LinkedIn where he feels out of the loop, I can already see us getting to-

Iain Morris: [00:59:59] He is out of the loop.

Scott Bicheno: [00:59:59] Well there we are. I can already see us getting to a point of automation and of AI-infused activity where that was, you said you don’t need that anymore in terms of problem solving. But this is what I query, and I don’t want to misquote you, so feel free to correct me. But if you get to a point where you’ve created something with AI and then you’re reliant on AI for the problem solving but let’s say for whatever reason, and this will be a segue to our next segment as well, the AI can’t solve that problem, there needs to be a redundancy, there needs to be a de-escalation where you can go, “All right, AI, you’re dropping the ball here, the big boys are going to step in.”

Iain Morris: [01:00:39] It’s another AI. It’s another AI.

Scott Bicheno: [01:00:39] No, no. A human being needs to be able to step in and go, “I’m f**king taking over here.”

Iain Morris: [01:00:43] You’re too obsessed with humans. Get rid of the humans.

Scott Bicheno: [01:00:47] Well, there we are. That’s kind of the point.

DR: [01:00:48] Yeah, I think you can ask, I saw it this summer with Benjamin and Scott, where Benjamin by the end of the summer would take a snapshot of the bug and send a picture to Cursor and say, “This is my bug, fix it.” And then he would pick up his phone and watch YouTube while it worked and then he’d come back and see it. And then with Scott, ex-husband, would be like, “Show me your code.” He wants to go into the code. And Benjamin just was like, “AI will fix it. I just need to go find it.” But you could just say, “Hey, AI, tell me about your code.” So if I needed to, the code’s still there. He’s still pull it open, right? It’s a lot.

Scott Bicheno: [01:01:27] But to the point that Iain illustrated about how this geezer reckoned, he couldn’t even get into the code and make sense of it directly.

Iain Morris: [01:01:35] I share that concern that what if something goes wrong. But even if it’s infallible, let’s say, I still think there’s a concern about relying on something that eventually, if we believe what Jensen Huang and people tell us, it’s going to outstrip human capability and we won’t understand it. And then it’s a case of what do people do actually, I guess, I mean we can’t have a world just of people who do prompting or something, let’s say.

DR: [01:02:01] Why not? We have a world of people who write code. I mean it’s the same thing. It’s just a different kind of code.

Iain Morris: [01:02:09] Yeah, I know. I just feel it raises a lot of… I mean this is very futuristic sounding, but if it’s a world where AI is able to do so many of the things that are done now by in white collar roles and let’s say robotics is capable of actually replacing a lot of other stuff. And there’s the questions about what people actually do that I think are different from early phases of automation, very different, I think.

DR: [01:02:29] Yeah, I think there was a World Economic Forum stat that yes, AI is going to eliminate a lot of jobs, but it’s going to create a lot of jobs more. And so I don’t think-

Iain Morris: [01:02:40] That’s my worry is whether it will.

DR: [01:02:41] I don’t think we’re going to have zombie world of people who don’t have jobs or what was that Ready Player One movie where no one was working.

Scott Bicheno: [01:02:49] VR and stuff.

DR: [01:02:51] And so everyone was living in the VR world and they didn’t have work. And I think there’s been lots of technological advances, humans persevere.

Iain Morris: [01:02:59] Yeah, we talk about this on the pod a lot, but it seems like this one’s quite different because it’s replacing a lot of the cognitive stuff that people have gone into when they haven’t been doing manual work. And another thing, we are going to have to move on something at some point, but I was going to ask you taking a step back from that, how you feel it’s going so far in telco specific, because from my observations, there’s a lot of talk about AI and a lot of what it could do. But I was looking at China Mobiles numbers the other day for a bit of a deep dive. And they’re regarded as this incredibly advanced sophisticated operator that’s even talking about level four, level five by something like 2030 and their headcount is 450,000 people.

[01:03:42] Now, okay, you could say that’s a big country and then lots of people to work in stores and everything. But in 2019, it was less. So it’s gone up over these six years of them probably starting to dabble in AI and everything. And I feel a lot of these companies that have cut jobs, they’ve not really realized cost benefits from it. You see the margins are still quite compressed. And wondering are telcos really benefiting from anything they’re doing in automation at the moment or AI?

DR: [01:04:08] I haven’t really seen it. I don’t know about the network side in terms of the automation, but I haven’t seen them deploying it at scale. Really, everyone’s still talking about this 2030. It’s still aspirational, it’s still in the future. So we’ll see. I don’t know if they’re going to cut jobs or ship. We have fewer people in customer support, but more people in another area. You don’t roll out AI just across the company all at once. And like I was talking about earlier, I’m like going from zero to one where you’re getting something working right is really hard. And even a small company like Totogi, one group of my organization adopted it readily, no issues. Another one, I have to smack it. I’m like, “Did you use AI for this?” And they’re just like, “You can’t use AI for that.” I’m like, “Oh.” Right. And we’re small and-

Scott Bicheno: [01:05:00] You’d love employing me and Iain.  You’d get some real f**king resistance.

DR: [01:05:03] No, I’d be like, “What is your deal?” But I’ll tell you another thing on the flip side of that coin is the people who’ve started to use it, if I said, “I’m going to take it away from you now.” They would be like, “No way. This is amazing.”

Scott Bicheno: [01:05:16] No, I get it. And I get some of it is generational. Well-

Iain Morris: [01:05:19] It’s addictive.

DR: [01:05:22] It’s so amazing.

Scott Bicheno: [01:05:22] But it’s what you’re used to. This is a generational cliche where all young people are all into this these days when you’re 54 like I am.

Pierre: [01:05:29] You thought the iPad was not going to catch on.

Scott Bicheno: [01:05:32] Yeah, exactly. My big prediction when they launched the iPads, it is just a big iPod. What’s the f**king point of that?

Iain Morris: [01:05:38] It’s just a big iPod.

DR: [01:05:38] Like hello?

Scott Bicheno: [01:05:41] Where it was a failure of my imagination was all the other ways. I mean especially when you look at it in retail and stuff. Anyway, but my kid’s generation, my son’s done a course. It certainly doesn’t sound as high-flying as your son’s course. And he’s certainly not coming home telling me he’s turning down $200,000 jobs.

DR: [01:05:57] He should train at Gauntlet AI.

Scott Bicheno: [01:05:59] Yeah, I know. I might have a word. Get him to listen to this pod. I go, “Look, Danielle’s son’s doing it. What’s wrong with you?” But he’s still doing stuff. And actually to the point of layers of abstraction, he’s learning Python, which is a layer of abstraction above all the coding stuff that you know. So yes, I get it with successive generations, to coin a term that you used and raise event, there’ll be more AI native as they grow up. Whereas people like us, and we are writers, so you’re going to get a bit more resistance from us. I was half joking about Grammarly. We are a bit precious about the words that come out of our brilliant minds. But it’ll become more endemic. It’ll become more baked in. Just like for when I was at school, computers were pretty basic, weren’t they? So I can see all that.

DR: [01:06:40] I mean, again, with writing, especially because I do a lot of writing, I’m not as prolific as you guys are, but I write a blog every two weeks. And so I’ll start with, “I know what I want to say and if you understand my style, AI, let me give you some bullet points of the facts of the story.” Like I’m going to talk about blah, blah, blah, blah, blah. And it pumps out a version that’s 90% what you probably would’ve done. And then you go from there and you’re just-

Scott Bicheno: [01:07:05] The purest in me pushes back even on that.

DR: [01:07:06] But why? It’s still you.

Scott Bicheno: [01:07:08] But it’s not.

DR: [01:07:09] It’s your take. It’s your angle.

Scott Bicheno: [01:07:10] It’s me tweaking something that something else has done. And I know you are saying, but it’s derived from me and my-

DR: [01:07:16] You let it. It is you.

Scott Bicheno: [01:07:17] Yeah. I think what I’m really exhibiting here as much as anything else is my own cognitive inertia about the whole f**king thing. I’m just like, no. But anyway, look, if you don’t mind, because we’ve got a couple of other things we want to talk about and we got a hard stop in about half an hour or so.

DR: [01:07:31] Okay, cool.

Scott Bicheno: [01:07:31] But talking about automated things going wrong.

DR: [01:07:34] Your favorite.

Scott Bicheno: [01:07:35] So yeah, I know it doesn’t time fly Iain.

DR: [01:07:37] That was fast.

Scott Bicheno: [01:07:38] Do you need another beer or are you all right?

DR: [01:07:40] No, no, no. I’m fine.

Scott Bicheno: [01:07:41] Okay.

DR: [01:07:42] I mean, you have your computer, huh?

Iain Morris: [01:07:42] Gin and Tonic?

DR: [01:07:43] Gin and Tonic, no, old-fashioned?

Scott Bicheno: [01:07:46] Yeah.

DR: [01:07:46] No. My beer is actually quite-

Iain Morris: [01:07:47] We do have a super strength whiskey from your part of the world.

DR: [01:07:52] Oh my gosh.

Iain Morris: [01:07:55] If you want one.

Pierre: [01:07:55] From Texas, we’ll show you.

DR: [01:07:55] From Texas?

Iain Morris: [01:07:56] It’s called Brimstone. Do you know that?

DR: [01:07:59] Oh, Brimstone. Oh my gosh.

Iain Morris: [01:08:00] Bit of ice?

DR: [01:08:00] That’s a terrible name.

Scott Bicheno: [01:08:01] The guy from EXA Infrastructure bought that in.

DR: [01:08:01] From…

Iain Morris: [01:08:05] Jim Fagan from EXA Infrastructure. They do fiber networks.

DR: [01:08:06] Okay. Yeah. Yeah.

Scott Bicheno: [01:08:10] Obviously I’m not expecting you to start necking whiskey. That could be quite entertaining if you did. But yes. So I got to say, actually.

Iain Morris: [01:08:18] There’s this paperwork coming out.

Scott Bicheno: [01:08:20] Look at her. She’s prepared.

DR: [01:08:21] A little prepared.

Scott Bicheno: [01:08:23] I thought in this last segment we did, you did a really good job, not that I  expected anything less of defending the position. And you’ve always been a great person speaking for myself, but I suspect Iain as well too, to stress test our skepticism against because you are well-prepared, obviously thanks in no small part to Lindsay, I’m sure.

DR: [01:08:42] Yes.

Scott Bicheno: [01:08:43] And you know your stuff because a coder and all that sort of thing. And in the past, we always used to have you as this, as we said before, this public cloud evangelist. And a lot of our concerns were similar, which is an over reliance on automation, over reliance on any third party. And anyway, this week there was just tailor-made ammunition for our-

DR: [01:09:04] You were like salivating.

Iain Morris: [01:09:05] It was ideal timing, wasn’t it?

DR: [01:09:07] It was.

Scott Bicheno: [01:09:08] So I’ll quickly remind everyone. I’m sure this was like, this is so big. This is front-page news on just regular news. I listen to Planet Rock while I’m working during the day and they have this tiny little compressed s**tty news bulletin once an hour. And it was even the front of the Planet Rock f**king news bulletin. And anyway, so it’s AWS, Amazon-

DR: [01:09:26] Well, I’ll tell you why. Well keep going.

Scott Bicheno: [01:09:28] So I’ll quickly summarize it and then I’ll pass it to you. So AWS stands for Amazon Web Services, the biggest public cloud provider in the world. And-

Iain Morris: [01:09:36] Not as big as they used to be though.

Scott Bicheno: [01:09:37] No, no, but they still are, I think a good chunk of market share ahead of-

Iain Morris: [01:09:42] Only in terms of market share. They’re actually-

DR: [01:09:42] Yeah. They’ve slowed down, I think they’re bigger. They’re very huge.

Scott Bicheno: [01:09:46] Ahead of Azure second and Google third, I think.

Iain Morris: [01:09:49] Yeah, there’s a bunch of neoclouds now. Aren’t there as well coming along?

DR: [01:09:52] Well, the AI stuff. The AI data centers like Oracle and stuff. Yeah.

Scott Bicheno: [01:09:56] Yeah. Oracle’s doing well.

Iain Morris: [01:09:57] Are you skeptical about Oracle?

Scott Bicheno: [01:09:59] Let me finish this first.

Iain Morris: [01:09:59] Sorry.

Scott Bicheno: [01:10:00] Jesus Christ.

DR: [01:10:01] I mean, we could have a four-hour pod.

Scott Bicheno: [01:10:03] You wondering why it takes us two hours to do a pod? There’s an illustration, especially after we’ve had a couple of beers.

Iain Morris: [01:10:08] We’ll go into Oracle later.

Scott Bicheno: [01:10:09] Yeah, we’ll get into Oracle. So let me quickly tee this up. A hell of a lot of people rely on AWS to run their digital infrastructure and they had an outage in their US-EAST-1 region or something like that. And it led to just loads of websites going down. Most of them, they sorted it out at their end within about a couple of hours.

DR: [01:10:28] Yeah, it was a DNS error.

Scott Bicheno: [01:10:30] Yeah. Yeah. Well, I wrote that up this morning. The actual specifics of it, in fact, I’ll come to that because this is one that really leans into my prejudices. So my headline this morning was “AWS blames massive outage on dot-dot-dot automation.” And so they had these things go wrong, which I’m sure you know all about. But anyway, long and short of it is a lot of websites went down. For some reason, Telecoms.com and Light Reading seem to be particularly badly affected, without slagging off my employer, it looks like we need to build in a bit more redundancy at our end. And we were out for most of the day, weren’t we? And anyway, so that’s it, I’ll just finish there without getting to the nuts and bolts of it. There is an example of being over reliant on a third party, even a very good, and I’m not saying AWS, It’s not very good. It is got where it is on merit, but even a very good one, there’s an example of a massive downside of that. What you got to say to that?

DR: [01:11:21] Yeah, I have a lot to say about that. And I wrote a blog about it in 2022 actually called, “Public Cloud Outages are your problem.”

Scott Bicheno: [01:11:31] Oh. Oh, you’re putting it back on me, are you?

DR: [01:11:33] Well…

Iain Morris: [01:11:34] It was your fault, Scott. I knew there was so-

Scott Bicheno: [01:11:36] Oh, I should known. Of course, it was my fault.

DR: [01:11:38] Now I imagine-

Scott Bicheno: [01:11:39] It’s always my fault.

DR: [01:11:40] I imagine your websites are through providers and we’ll talk about that.

Scott Bicheno: [01:11:44] Yes, I’m sure.

DR: [01:11:45] US-EAST-1 is…

Iain Morris: [01:11:48] It’s pretty old, isn’t it now?

DR: [01:11:49] It’s the oldest. It was the first, it’s the oldest, it’s the biggest and it’s the first place they put any new changes. And because of that risk, it is the cheapest for all the services. So a lot of startups and companies when you’re just putting your services together, you’ll put your stuff in US-EAST-1 because it’s cheap.

Scott Bicheno: [01:12:10] Right. It didn’t even occur to me that there was different pricing between different zones, right.

DR: [01:12:13] There is. Right? For example, DynamoDB, which was a service affected is what Totogi’s charging as a service uses. We use DynamoDB, but I don’t have my stuff in US-EAST-1 because charging cannot go down. That would be detrimental to Totogi.

Scott Bicheno: [01:12:27] So you are saying that a high degree of risk is priced into US-EAST-1.

DR: [01:12:32] Correct.

Scott Bicheno: [01:12:32] Right.

Iain Morris: [01:12:32] But I also heard that US-EAST-1 is where a lot of common stuff happens to do with the control plane.

DR: [01:12:38] Correct.

Iain Morris: [01:12:38] Therefore, you almost can’t avoid it-

DR: [01:12:40] But you can.

Iain Morris: [01:12:40… unless they’ve done it like a European sovereign cloud type thing.

DR: [01:12:42] Yeah, to Boost Mobile in your article.

Iain Morris: [01:12:43] Well, Boost Mobile went down though.

DR: [01:12:45] Yeah, let’s talk about that. So Boost Mobile went down because they rely on a service called, I guess, was it S3 or it was, or Route 53. It’s their DNS routing.

Iain Morris: [01:12:55] Peter Adderton didn’t like my article.

DR: [01:12:56] Oh, okay, yeah.

Iain Morris: [01:12:56] The founder of Boost Mobile, He had a massive go at me on LinkedIn.

DR: [01:12:56] Oh, really?

Iain Morris: [01:12:56] Yeah.

DR: [01:13:02] On LinkedIn. Oh, interesting.

Iain Morris: [01:13:02] Yeah.

DR: [01:13:03] I follow him on Twitter. He’s very-

Scott Bicheno: [01:13:06] So that wasn’t a message, that means he was publicly having a go at you.

Iain Morris: [01:13:08] It was in a, yeah, like a comment-

Scott Bicheno: [01:13:09] In the comments, right.

DR: [01:13:12] I didn’t realize Boost went down and I researched it on the way to come talk to you.

Iain Morris: [01:13:16] I didn’t know until I started wondering if they went down and then they admitted that they got down.

DR: [01:13:20] They did.

Scott Bicheno: [01:13:21] And he got defensive.

Iain Morris: [01:13:21] But I got in touch with Downdetector who said that they had 5,000 complaints that day that they couldn’t get any signal at all, but about 25,000 about the service. They’ve got 7 million subscribers but I think most of those are on AT&T and T-Mobile because their network is, they’re mainly-

DR: [01:13:39] They have the multiple networks.

Iain Morris: [01:13:40] I guess their whole network was not really working.

DR: [01:13:43] Like I wrote in my blog, you can spend more money to build more resilience into your product, right? Amazon has four major regions in the United States. Each region has at least three availability zones, and an availability zone is a data center. They’re usually about at least 50 miles apart. So you can have a bit-

Scott Bicheno: [01:14:06] So if there’s a natural disaster or something that-

DR: [01:14:07 Correct. Will a flood happen, you know, everywhere? Maybe. Unless it’s like Noah times, maybe not. But so you can do stuff with your setup at Amazon to fail over within region to another availability zone or to another region, right? So you can have cross regional and with more resiliency is more cost. So how resilient do you want to be? So if you’re a little startup, little tiny maybe web hosting thing that’s not that big, you’re optimizing on cost. You’re going to put it into US-EAST-1 and you’re going to take the risk with the trade-off of the lower prices.

Iain Morris: [01:14:50] Or Light Reading and Telecoms.com.

DR: [01:14:50] Right?

Scott Bicheno: [01:14:50] Apparently.

DR: [01:14:50] So that’s what they’re doing. You could sit there and tell your provider, “Move literally to any other data center.” It probably won’t be that much more expensive, but get out of US-EAST-1 and you won’t take the hit. That’s almost always where there’s outages for AWS. If you go back four or five years, maybe even 10 years, always US-EAST-1.

Iain Morris: [01:15:09] Smart CIOs have basically gone-

DR: [01:15:11] Don’t put in.

Iain Morris: [01:15:12] This US-EAST-1 area, we want to pay a bit more and actually ensure that it’s hosted.

DR: [01:15:15] I have stuff in-

Iain Morris: [01:15:17] Can you avoid that control play in common-

DR: [01:15:19] Yes, you can. With Boost Mobile, they could have backup redundancy in another region for that same service to automatically fail over. They might have a little bit more latency for the East Coast people, but you’re up. It’s just a little bit slower and then you fail back over.

Iain Morris: [01:15:36] They obviously haven’t done that though.

DR: [01:15:37] They haven’t done it, and probably because they’re small and they were struggling with all their money.

Iain Morris: [01:15:41] And they’re also giving up on their network now anyway. They’re not going to invest in it now.

DR: [01:15:45] What’s the point?

Scott Bicheno: [01:15:48] Well, he’s not going to like that either. You’ll get more LinkedIn aggro off him for saying that.

Iain Morris: [01:15:55] I mean, he was the guy who founded it as an MVNO.

DR: [01:15:55] Right, and then it got bought.

Iain Morris: [01:15:55] Bought and turned into a network.

Scott Bicheno: [01:15:56] Given he’s obviously feeling a bit defensive, I was just going to-

DR: [01:15:58] Right. I wrote this great blog that I’m probably going to-

Scott Bicheno: [01:16:01] Even if you say so yourself.

DR: [01:16:02] Right. So I did not write this with AI. This is like 100% DR creation.

Iain Morris: [01:16:06] You didn’t have the option then.

DR: [01:16:07] It’s probably sucky to be honest.

Scott Bicheno: [01:16:10] On the contrary, it’s full of your personality.

DR: [01:16:12] I wrote it with a quill and a-

Scott Bicheno: [01:16:13] And your unique fingerprint for it.

DR: [01:16:14] And ink back in the old days. But yeah, it’s a shared responsibility for resiliency, right?

Scott Bicheno: [01:16:21] Yeah, that’s a good point.

DR: [01:16:25] But then I’m going to flip it, right, Vodafone had an outage here last week.

Iain Morris: [01:16:30] They did.

DR: [01:16:30] And they blamed it on a third party vendor upgrade, so things are going to go down.

Iain Morris: [01:16:35] Well, that is exactly what Peter Adderton said on his LinkedIn thing. He said, this is silly because any network can go down and exactly like Vodafone went down and it was really disruptive for loads of businesses. And I take that point, I guess if I was going to push-

DR: [01:16:48] Telefónica was down in the summer, right?

Iain Morris: [01:16:50] They were. I guess if I was going to push back on that, I’d say when a private cloud goes down or an on-premises thing, it just affects that company. And if it’s a network serving millions of people, it’s still bad. But if AT&T, let’s say and Verizon and T-Mobile had done what Boost Mobile’s done and gone, “Let’s rely on AWS,” then it would’ve been really catastrophic for like comms in North America.

DR: [01:17:13] I was interacting with AI and I was like, how long would it take Boost Mobile to do this redundancy? Assuming you’re still with an AWS heavy strategy, given that right now clearly they’re in US-EAST-1, but let’s set up resiliency, three to five business days.

Iain Morris: [01:17:29] Right.

Scott Bicheno: [01:17:30] It’s an interesting thing actually, when I wrote my thing, was that this morning? Yeah. My thing about automation confirming my prejudices as I admit, and I said my final sentence in it, and this is interesting, I want you to tell me, even allowing for the fact that if you don’t mind me saying it, you have a bias in favor of the public cloud.

DR: [01:17:50] Sure.

Scott Bicheno: [01:17:51] Whereas I wrote, “This is probably a good time to be in the redundant private cloud business.” Now, I don’t know what the f**k I’m talking about, I’m just a journalist.

[01:17:58] But the reason I thought of that, we obviously know about the hybrid cloud, which is hedging your bets a little bit. And I just thought the one thing you can really trust, because what you described, I didn’t know about these different regions and these different things, so that’s really interesting to know. But what you described is still for me a slightly, I don’t know, jarring sales pitch. We’re like, “You can have this but it might be s**t so pay us more money and you can have a backup in case this one,” it just feels like a slightly counterintuitive sales pitch. Obviously, the way I’m positioning it isn’t the way they position it. But the ultimate one in theory you can trust, although I’m sure you’ve got objections to that, is to have some kind of redundant private cloud like local infrastructure.

DR: [01:18:43] Well, I mean you can do that, but I think your cost goes up, right?

Scott Bicheno: [01:18:43] Yeah, exactly. So you might as well do the, hedge your bets on AWS or whatever.

DR: [01:18:45] Yeah. I mean-

Scott Bicheno: [01:18:46] Would that be cheaper? Would it be cheaper to have redundancy within a different region of AWS rather than having your own private situation. Yeah.

DR: [01:18:53] For sure.

Scott Bicheno: [01:18:54] That makes sense.

DR: [01:18:55] Right? And especially just bringing it back to the AI thing, I see a lot of telcos wanting to build their own AI data center on prem for sovereignty reasons or whatever. We had a telco do an estimate. They asked us how much would it cost for us to set up our own AI data center to do whatever they were doing and our estimate was $185 million.

Scott Bicheno: [01:19:17] Oh my God.

DR: [01:19:17] And so I was like, “Okay, let’s just take that number.”

[01:19:23] But in this scenario, this telco in the Middle East wants to start a new business. They want to use heavy AI, they want to do all these crazy things and they want to host their own LLM and they want their own GPUs.

[01:19:36] And I’m like, “You have so many variables on this brand new business, you don’t know if it’s even going to work.”

Iain Morris: [01:19:42] But that’s quite an extreme-

Scott Bicheno: [01:19:43] Why over invest in something that, yeah.

Iain Morris: [01:19:44] That’s quite an extreme vision-

DR: [01:19:45] Why don’t you use the public cloud? Why don’t you use the public cloud?

Iain Morris: [01:19:48] Yeah, I think-

DR: [01:19:48] And figure out if it’s going to work, and if it does and it’s great and it’s a wild success and you still want to do your own hosting of your LLM and your GPUs, then do it at a later date.

Iain Morris: [01:19:59] I think what they’ve done is have a very extreme interpretation of sovereignty, because Europe’s vision of sovereignty, it doesn’t necessarily mean you don’t use AWS. I think at the moment-

DR: [01:20:10] It’s just in country.

Iain Morris: [01:20:12]… you can use AWS, but it’s where the control plane sits for the European sovereign cloud with Amazon I think’s in Germany, which suits people like Telefónica Germany and the EU. And then there’s now an argument about do we actually want US companies involved or should it be European providers? But that’s more of a geopolitics thing.

[01:20:28] But I think there’s got to be a middle ground between, I don’t actually limit this to the public cloud by the way. I just think this is a really sensible pushback I think to the argument that, oh, we shouldn’t use AWS because we’re too reliant on a few tech providers. I think these oligopolies are there in a few industries to do with tech that people just don’t really realize. Broadband infrastructure is only really provided by Nokia at scale these days if you take out the Chinese companies and people don’t really-

DR: [01:20:57] Question.

Iain Morris: [01:20:57] If you say to somebody, “I’m off to meet Nokia, ” as I did to some friends recently.

[01:21:01] They were like, “Are they still in business? They’re the guys who did those phones, aren’t they? Before Apple came.”

DR: [01:21:05] Right.

Iain Morris: [01:21:05] It’s like all your network fiber infrastructure in the UK-

DR: [01:21:08] Is Nokia.

Iain Morris: [01:21:08]… pretty much comes from it because the only options otherwise are Adtran and Calix and they have 3% of the market. But the cloud ones just seems like a more worrying version of that because so much data’s going into it so much of our-

Scott Bicheno: [01:21:22] Yeah. And we realized this Monday, so much relies on it.

Iain Morris: [01:21:25] So much relies on it in terms of people are talking about connectivity as well as just information lakes and this sort of thing.

Pierre: [01:21:34] There’s national security at some point.

Iain Morris: [01:21:35] And I guess that’s why you get this conversation going on at the UK at the moment about the cloud services. Is it too dominated by Microsoft? And that’s propelled these companies like Nscale into the, I don’t know what you think about those guys by the way.

DR: [01:21:50] Well, I mean I guess it’s your risk tolerance and whatnot, but they’re really good at running infrastructure, right? And from a purely capitalist view of it, of where would you rather spend your resources and time and money and people on your business when it’s a commodity thing to run servers and compute and get your chips when you’re trying to solve all these other problems. And sure, it’s a concern, but at the same time the alternative is to go do it all yourself and now you’re focusing on that and your, let’s say the guy-

Iain Morris: [01:22:26] And they’re never going to be able to be as good.

DR: [01:22:28] Totally, right?

Iain Morris: [01:22:29] I get that completely and have all those services that you get from Amazon.

DR: [01:22:31] It’s a business decision at the board level of what’s our special sauce? And I think the ship has sailed that special sauce is running servers. You’re like, “Just let these guys do it.”

Scott Bicheno: [01:22:40] In a way, it’s analogous to the AI thing. It’s like you’re offloading as much as possible to focus on your special sauce or core competence or whatever we want to call it. And that all makes sense. I mean I suppose, I’m trying to think of it. At our day job, I suppose if we could outsource research and just go, if you sat down, like Iain, as I’m sure you’re aware, writes more in-depth stories than I do.

[01:23:03] If he sat down in the morning and go, “Right, this is my theme, this is where I’m going to go. I’m going to go and make a coffee, I’m going to send this prompt to the AI and go just give me a f**king bunch of condensed research on this for me to cherry-pick from,” and then it’s still Iain’s stuff, but it’s done a lot of the legwork of the research for you. I can see that, but then even then that might feel like a slippery slope.

Iain Morris: [01:23:22] I mean, I’m almost on the other slippery slope now where I feel like you are almost overly rely on search engines for things. Especially when Gemini throw this up-

DR: [01:23:32] Oh gosh. you guys.

Iain Morris: [01:23:34] I know, I worry about this in a big-

DR: [01:23:35] You’re like, “It doesn’t count unless I go to the library.”

Scott Bicheno: [01:23:36] To the British library and go through parchments.

DR: [01:23:38] You are not into the Dewey Decimal System, you are not a writer.

Pierre: [01:23:43] He uses-

DR: [01:23:44] I mean that’s a little bit extreme.

Pierre: [01:23:45] … those scanners with a lamp.

DR: [01:23:45] Microfiche.

Scott Bicheno: [01:23:45] Microfilm. Yeah, we got it. Microfilm, yeah.

DR: [01:23:45] He’s sitting there, bzz, bzz, bzz, bzz. There it is. There it is. It’s the headcount of China Mobile.

Scott Bicheno: [01:23:54] But you’re right.

DR: [01:23:55] From yesterday’s newspaper.

Scott Bicheno: [01:23:57] You’re right, I mean that’s a great illustration. All Iain’s doing, as I’ve been, is being honest about our cognitive resistance to this stuff. But I still-

DR: [01:24:07] I’m going to argue it’s still you.

Scott Bicheno: [01:24:10] No, and what you’re really good at and where we’ve pushed back on you in the past, but credit where it’s due. What you’re really good at is seeing what’s coming. To the old Gretzky things, skate to where the puck’s going to be.

DR: [01:24:19] Skate to where the puck will be.

Scott Bicheno: [01:24:20] And fair play to you on that.

DR: [01:24:22] And that is my job, right? Making calls and-

Scott Bicheno: [01:24:25] Yeah.

Iain Morris: [01:24:26] I was going to ask because I’ll forget otherwise, but the BSS Magic thing just quickly, that’s hosted with AWS, you couldn’t put that on-premise and do it that way.

DR: [01:24:34] We could if we… The end result could be deployed wherever you want. So what we did this week, we were just shipping to AWS because that’s super easy to do. But if a telco’s like, “Hey, ship to our production environment, it’s on-prem,” no problem.

Scott Bicheno: [01:24:49] Okay.

DR: [01:24:50] So super. We’re not forcing that on you.

Scott Bicheno: [01:24:53] To round off the public cloud thing, and if you’ve got any public cloud tangents, you want to mention them, feel free. Your point to your blog that you wrote that I can see because you printed it out so I can see it.

DR: [01:25:02] I did. I brought three copies.

Scott Bicheno: [01:25:03] All right, cool. Yeah.

DR: [01:25:0] Everyone.

Scott Bicheno: [01:25:05] I’ll grab one.

DR: [01:25:06] Yeah.

Scott Bicheno: [01:25:06] Or we can put one in the middle there.

DR: [01:25:07] I will link it in my next, probably in the show notes.

Scott Bicheno: [01:25:10] Where you said public cloud outages, I quote your problem or you put asterisks, I don’t know what significance.

DR: [01:25:15] Well, I’m emphasizing your problem.

Scott Bicheno: [01:25:17] I see, yeah.

DR: [01:25:17] You need to go-

Iain Morris: [01:25:18] 2022, wow. That was pre-generative AI almost.

Scott Bicheno: [01:25:20] Yeah, exactly. So there we are.

DR: [01:25:22] And you’ll notice, from Iain Morris the first thing, for example after Google outage, telcos should think-

Scott Bicheno: [01:25:30] Oh, yeah, and then I got the second one.

DR: [01:25:32] You got the second one.

Scott Bicheno: [01:25:33] Cool.

DR: [01:25:33] After Google outage-

Scott Bicheno: [01:25:34] I see that.

DR: [01:25:35] … telcos should think twice about public cloud, that’s from Iain. And then from Scott, the public cloud wilts in the UK.

Iain Morris: [01:25:42] At least I’m persistent, I’m saying the same thing nowadays, aren’t I?

Scott Bicheno: [01:25:44] God, my headline there seems more like an Iain Morris headline.

DR: [01:25:48] The minute that you guys were like, “Oh, we’re going to talk about the AWS outage.”

[01:25:51] I’m like, “Pull up that blog from 2022.”

Scott Bicheno: [01:25:54] So what I’ve taken from that is there are lots of tools within the public cloud to do with redundancy. That’s the biggest thing that occurs to me. Again, I’m not an expert, I’m not like you, I’m just a commentator, but I understand the concept of redundancy and I especially understand the concept redundancy when my f**king website isn’t up for half the day and it can be done. But as you say, it’s a matter of resource.

Iain Morris: [01:26:14] I wasn’t worried about that at all.

Scott Bicheno: [01:26:16] Iain was fuming.

DR: [01:26:17] Well, I mean again.

Scott Bicheno: [01:26:17] It’s a matter of resource and you are saying the more it’s-

DR: [01:26:20] And it’s not that much more money to move it to Oregon or Ireland or whatever. It’s not.

Iain Morris: [01:26:26] There aren’t that many of the regions are though, in the European-

DR: [01:26:29] In the US there’s four.

Iain Morris: [01:26:29] There’s only four.

DR: [01:26:30] But I think there’s-

Iain Morris: [01:26:31] There’s a couple of, that only the government can use.

DR: [01:26:33] There’s 36 around the world.

Iain Morris: [01:26:34] There’s some weird ones that only the government use.

Scott Bicheno: [01:26:36] There’s one in Frankfurt, isn’t there?

DR: [01:26:37] Well, there’s six in the United States.

Iain Morris: [01:26:39] There’s not very many. There’s only one in Germany.

DR: [01:26:44] Two are GovClouds, and they’re super expensive.

Iain Morris: [01:26:45] GovClouds, that’s what they’re called.

DR: [01:26:45] They’re super expensive.

Iain Morris: [01:26:46] You can use GovCloud as a normal organization.

DR: [01:26:49] If you want super US people only and it’s in a cage and all these rules, it’s security clearance, blah blah blah. You can totally use it. You’re just going to spend more, right. Just do you need that?

Iain Morris: [01:27:01] The thing I was surprised by is that they don’t have as many local zones as I thought they’d have. They don’t have that many local zones, which I-

DR: [01:27:08] No, I can’t remember.

Iain Morris: [01:27:08] This is the problem I think with, because this isn’t a security or redundancy issue. This one is when you’re putting with the network in the cloud, there are things you just need where you need it to be at the edge.

DR: [01:27:22] Yeah, and that’s for-

Iain Morris: [01:27:22] And unfortunately with AWS infrastructure, and this is why they’ve had this compromise where they’ve gone, we’ll take our outpost servers and we’ll put it in your infrastructure. But then obviously you lose a lot of the economies that you get of shared cloud infrastructure in a big kind of data center. I believe on the BSS/OSS side from a service point of view, but on the putting your RAN there, I mean that’s just crazy. No one’s doing that.

DR: [01:27:45] So that’s where I would tell you to spend time with Mallik Rao and go see what he’s doing with Telefónica Germany because he’s working with Nokia. Nokia I think is actually how you say it.

Iain Morris: [01:27:54] We say Nokia in England.

DR: [01:27:56] Yeah. And AWS.

Iain Morris: [01:27:59] No, I know Mallik, I speak to him at MWC usually sometimes.

DR: [01:28:01] Right. Talk to him about how AWS has worked to extend and improve local zones to meet him because he is planning to deploy, he is deploying his core on the public cloud on AWS. He’s starting with 5 million subs. But I think his plan is to do them all. And he was on our podcast, my podcast a couple of months ago talking about how AWS needed to understand telco requirements and bend over to do it and they modified it and it’s working. So to your point, it will work. Maybe not out of the box. Maybe not out of the box that was designed in 2015. But they’re willing to do that and they’re doing that for Germany.

Scott Bicheno: [01:28:41] Saw you beavering around Perplexity there on your-

DR: [01:28:44] I love Perplexity, why is-

Scott Bicheno: [01:28:45] Yeah. Did you want, before I move it on-

DR: [01:28:47] I was going to pull up how many.

Scott Bicheno: [01:28:47] Is there something you were going to look at?

DR: [01:28:48] I was going to look up how many local zones there are because I don’t have it off the top of my head.

Iain Morris: [01:28:53] Two.

Scott Bicheno: [01:28:53] But that’s one thing I’ve taken. Yeah, f**k it, that’ll do. One thing I’ve taken away is that that redundancy thing is there. It’s just a matter of resource and I suppose to have a bit of a go at our own IT people, don’t take this personally, but one possible interpretation of the fact that our sites were down for so long on Monday was that we were under-resourced on redundancy. But again, I don’t really know.

Iain Morris: [01:29:16] I can’t imagine that would ever happen, Scott.

Scott Bicheno: [01:29:17] No.

DR: [01:29:18] Just move to another region.

Scott Bicheno: [01:29:20] Yeah, just do that. All right?

DR: [01:29:21] Yeah, it should be fairly easy.

Scott Bicheno: [01:29:24] Cool.

DR: [01:29:25] But I think that’s what happens. I think startups go in US-EAST- 1 because it’s cheap, then they start to scale. Then it’s like, “Okay, now we need to move it.” Which is a project. It’s not like just push, I mean it is a little bit of a push a button, but if depends on how you set it up. And it’s-

Scott Bicheno: [01:29:37] Do you think a risk of not being able to move on, but it’s a big thing.

DR: [01:29:40] For example-

Scott Bicheno: [01:29:41] You think the public cloud people could make that a bit easier? I mean, it’s in their interest.

DR: [01:29:45] Well, it’s all in your design.

Scott Bicheno: [01:29:46] Okay, fine.

DR: [01:29:47] How you architect it. But for example, I remember Netflix used to go down and now they don’t, they’re not on that list anymore and it’s just people moving out of US-EAST-1 to build resiliency.

Scott Bicheno: [01:29:57] That’s really interesting. I didn’t know that. So that’s one of those-

DR: [01:29:57] I think this is the best podcast ever right now. It’s not just the beer talking.

Scott Bicheno: [01:30:03] No, it’s great. I mean, this is stuff we’re all interested in.

DR: [01:30:05] Yeah, it’s super cool.

Scott Bicheno: [01:30:07] And you could go on forever. I always like to think the pod should just be like if you stuck a microphone on the table in the pub and you’re just having a chat.

DR: [01:30:15] Yeah, that’s what it’s like.

Scott Bicheno: [01:30:17] And maybe if you’ve got time after we can put that to the test.

DR: [01:30:19] Okay, cool. Yeah, for sure.

Scott Bicheno: [01:30:22] I was going to ask you if we can keep this short, how bubbly do you think the AI thing, not from the point of view of your day job. You’ve done a great job of advocating on behalf of AI in the telco context, but just more generally these under billion here, 500 billion there sort of deals. How bubbly do you think the whole AI sector is right now?

DR: [01:30:42] I don’t know. I mean a little bubbly, right? I mean especially that really great Bloomberg graphic. I don’t know if you guys saw it, of all the money that’s very circular.

Scott Bicheno: [01:30:51] The circular bit is fascinating.

DR: [01:30:51] That feels a little bit dicey, at that level. Certainly not my level. It’s not a nothing burger. It’s a something burger. But maybe not as big as it is.

Scott Bicheno: [01:31:00] It gets a little bit hype curvy and it will level out.

Pierre: [01:31:04] We just don’t learn anything, we just want to keep doing it.

Scott Bicheno: [01:31:06] Because the incentives that are baked into bubbles and hype, I always equate it.

Pierre: [01:31:11] Not if you’re holding a bag.

Scott Bicheno: [01:31:11] I always equate it, yeah, to musical chairs. People, this sounds a bit bubbly, but it keeps going up so f**k it, I’m not going to bail out. I mean famously, someone like Warren Buffett bailed out the dot-com bubble curve early. But then the people who hung around who timed it perfectly, which is obviously a big if.

DR: [01:31:30] That’s hard to do that.

Scott Bicheno: [01:31:31] But the people who timed it perfectly were laughing.

DR: [01:31:33] Yeah.

Scott Bicheno: [01:31:33] So yeah. Okay. All right, look, we’ll wrap it up there. We’ve all got stuff to do. I saw you two exchanging glances as to whether you got time to join us from there.

DR: [01:31:42] Maybe a little bit.

Scott Bicheno: [01:31:43] You’d be very welcome. It’s entirely up to you. When are you going back to the US?

DR: [01:31:47] Tomorrow.

Scott Bicheno: [01:31:47] Tomorrow. Okay.

DR: [01:31:48] Yeah. I came back from the Germany event to do this.

Scott Bicheno: [01:31:51] Right.

DR: [01:31:51] Spend the night, and then there’s now two non-stops to Austin on BA, so.

Scott Bicheno: [01:31:56] Oh, direct to Austin?

DR: [01:31:57] Yeah.

Scott Bicheno: [01:31:57] Good.

DR: [01:31:57] Which is nice.

Scott Bicheno: [01:31:58] That’s handy, isn’t it?

Iain Morris: [01:31:58] That’s good.

DR: [01:31:58] I’ll land at 4: 00, right? I can play some pickleball if I want to. Yeah.

Scott Bicheno: [01:32:04] Pickleball, yeah,.

DR: [01:32:05] It’s my new thing, I’m going to Nationals.

Scott Bicheno: [01:32:06] I remember you talked about that before.

DR: [01:32:07] I won a tournament last weekend.

Scott Bicheno: [01:32:09] Oh, wow.

DR: [01:32:09] Going to Nationals in pickleball, so that’s great.

Scott Bicheno: [01:32:13] Is that what you do post-tennis?

DR: [01:32:14] Yeah, well I’m playing both, which is causing a little bit of brain racquet confusion.

Scott Bicheno: [01:32:19] Right, different techniques I presume.

Pierre: [01:32:21] I found the bounce on pickleball is really inconsistent.

DR: [01:32:23] Yeah, I mean it’s a different game, but I mean if you’re a decent tennis player, I mean you can just clean up with pickleball, but then at some level you actually have to learn the game and that’s, I’m at that transition.

Scott Bicheno: [01:32:35] Is it better to come into pickleball from tennis or from squash?

DR: [01:32:39] Ooh.

Scott Bicheno: [01:32:41] Because it’s got that squash.

DR: [01:32:41] Maybe tennis.

Scott Bicheno: [01:32:42] It’s got that squashy thing with a backboard.

Iain Morris: [01:32:45] No, that’s paddle you’re thinking of.

Scott Bicheno: [01:32:46] Oh paddle, sorry.

Iain Morris: [01:32:50] Paddle’s a bit more squash-like when it comes off the back board.

Pierre: [01:32:50] Pickleball is a large table tennis.

Scott Bicheno: [01:32:52] Right?

DR: [01:32:54] It’s like ping pong in tennis.

Iain Morris: [01:32:54] I played a lot of paddle in Slovakia.

Scott Bicheno: [01:32:55] Yeah, my bad. I was confusing the two.

Iain Morris: [01:32:56] You need four people with paddle as well-

Pierre: [01:32:58] And you only score when you’re serving, which is confusing.

DR: [01:32:59] It’s like old volleyball.

Scott Bicheno: [01:32:59] That’s it.

DR: [01:33:02] You play with the wiffle ball, you put it with a plastic ball with holes in it.

Pierre: [01:33:05] If it lands on the hole it doesn’t bound the same as it between holes. Yeah, whatever.

Scott Bicheno: [01:33:10] I read a funny article about paddle. You say it’s always doubles, yeah?

Iain Morris: [01:33:12] Yeah.

DR: [01:33:13] Yeah, same with pickle.

Scott Bicheno: [01:33:14] Someone wrote an article about paddle. They said the reason it’s so popular is because people play mixed doubles and it’s a great way of flirting with each other.

Iain Morris: [01:33:21] Oh yeah, totally. And the course-

Scott Bicheno: [01:33:23] It wasn’t even a surprise there.

Iain Morris: [01:33:24] The courts are 80 quid in London as well to get a paddle for one hour.

Scott Bicheno: [01:33:29] So it’s become the new place to pull.

DR: [01:33:29] You play with mixed.

Pierre: [01:33:30] You have two-

Scott Bicheno: [01:33:31] Maybe I’ll give it a go, only joking.

Iain Morris: [01:33:32] When you play with your mates who work for some company that Danielle owns and it’s like they earn enough money to pay 20 quid each. But if you take your kids out-

DR: [01:33:41] It’s expensive.

Iain Morris: [01:33:42] It’s 80 quid for one hour to take my five-year-old to teach him paddle. It’s like no chance.

DR: [01:33:47] Back to your point on, it’s a big place to flirt or whatever. When you’re playing mixed doubles, the guys are playing, they’re playing big across the middle. They’re playing 65% of the court and confining the women to 20, 25. I’m just like, “F you, I can play too.”

Iain Morris: [01:34:04] Yeah, there’s a lot of that.

DR: [01:34:04] But there’s a lot of, the guys cover.

Scott Bicheno: [01:34:06] You’ve got to find ways of just tripping off or something.

Pierre: [01:34:07] They’re mansplaining on the.

Scott Bicheno: [01:34:11] They’re pickle-splaining.

DR: [01:34:12] And then I’m better than them and it’s just so annoying.

Iain Morris: [01:34:15] Paddle-splaining.

Scott Bicheno: [01:34:17] Paddle. I keep mixing up. Okay, well-

Iain Morris: [01:34:17] I’ve never played pickleball, I’ve only played paddle.

DR: [01:34:19] It’s fun.

Scott Bicheno: [01:34:20] On that note, given that I obviously can’t distinguish between the two, we’ll wrap it up there.

DR: [01:34:23] It’s okay, it was fun.

Scott Bicheno: [01:34:24] Danielle, it was a pleasure again.

DR: [01:34:25] It was super fun.

Scott Bicheno: [01:34:26] And keep in touch next time you’re in town you’ve got an open invitation.

DR: [01:34:29] For sure.

Scott Bicheno: [01:34:30] Especially while you keep bringing us such excellent-

DR: [01:34:32] I know, it’s like great gifts.

Scott Bicheno: [01:34:33] We should make sure, I don’t know, Lindsay, if you’ve got a shot with these branded things. But we’ll do that. We’ll do that when we finish. But thanks a lot and I think we’ve just got the one next week. Make sure you join us for that one.