0:04 So when you ask it something, because it wants to please you, it gives you an answer, even if it's only 10 or 20% sure of the answer. In our current world model, you need innovation to be financially profitable. There's a data centre being built right now in America that will use as much power as the entire city of London. Just one data centre. Several more like it are going up around the planet.

0:28 AI is like a very intelligent child, I guess it is today. It wants to please you, so it never says I don't know, even when it's only 20% sure. It just tells you the answer with absolute authority. In this episode, we get into why that matters more than most people realise.

0:45 Why asking AI to convert a Word document to a PDF is a bit like phoning a £400 an hour lawyer to fetch you a pint of milk. And why running models locally on your own machine might be the most sustainable choice you can make for your wallet and for the planet. If you ever wonder what's happening. If you ever wonder what's actually happening behind the answer AI gives you, stay tuned.

1:05 Let's uncover what is better with humans. In this episode, I'm joined once again by my close friend, Leslie Coelho. Leslie is back by popular demand as we're receiving questions around using local or private AI models, growth and sustainability. Leslie spent 40 years in software engineering and IT and has lived through every major wave of technology.

1:26 From writing safety critical aviation code for the Airbus A321. No internet, no predefined code libraries and no AI to helping tesco.com during the dot-com boom and experiments with educating people on AI tools in his retirement today. So Leslie, welcome. Hi, Shaun.

1:45 It's good to be back. So I was thinking that, so some of the questions we've had have been around, as we said, using private LLM models. So we'll go and be onto that. So anybody that's listening, we will kind of expand on that.

2:00 We'll expand on some of the elements around that. Some of the issues around. Sustainability. Some of the reasons, reasons other than privacy that you might want to use local AI models at home. And, but before we start, I thought it'd be really good to actually expand a bit, a little bit more on what we talked on last time.

2:19 We did talk a bit about how AI models work so that they tokenise. But I thought it would be good to cover a bit more about actually how are AI models trained. So we look at it in layman's terms in terms of how they are trained. And why then that introduces some limitations and biases and other things like that.

2:39 And recently I was on some training and they used a great analogy of what they called a super librarian. So you imagine if in the, before we had the things like ChatGPT, you would have, you would have Google. And imagine the, this, this like super librarian is like, is, is, is Google. So somebody can go over to the library and say, Hey, I've got this thing.

3:02 I want this information. So they go off. And then as we see me, Google will come back and here are a list of, of links, information that you can go off and you can, you can research. And then if we then look at AI, we're imagining, okay, that super librarian is learned, is taken or is read all this information in this library.

3:19 Everything that's available in this massive library of, of on the internet and said, right, this is everything that we've got in our library. And then you take that library and you put them in a room, you put them in a room. And maybe all they've got is a, is a, is a piece of paper. So then you go to that library and you say, okay, right.

3:39 You ask it for some information. Now it's going to take the information from memory. What have I, what have I learned? What do I know about that?

3:48 Write on that piece of paper and then hand it back to the, to the person. Now let's, let's say six months later, we go back to that, to librarian and we ask it something that happened in the last week. Let's say there's a new president of the, of, of, of, you know, of the United States. For example, we ask it, who is the current president?

4:09 Then it's going to go off and what do I know? I actually, as far as I know, this isn't. So it's going to give us the, the result of the, the previous, previous president. So we have that limitation of time and the limitation on knowledge from a specific time.

4:27 So as we, as we, what do you, what do you, what do you think about that as an analogy? Would you, anything else? Would you add to that? What else would you, would you add?

4:34 I would say it's useful, but incomplete. So let's take a, let's change your analogy slightly and say, I'm interested in a particular topic. Let's say behavioural economics. And I go and find an expert on behavioural economics and I ask them questions.

4:51 That person is an expert. The problem you have with AI is that it's fed huge amounts of information. So if you're, if you were training a model, a small model on specific information, you may have the opportunity to curate that information and make sure you only give it trustworthy sources.

5:08 So if it's behavioural economics, you probably want to take experts in that field, like Tversky and Kahneman, and not just Fred from down the pub. Because of the huge amounts of information that's been ingested by AI, nobody bothered to curate it. So it's actually kind of a who shouts loudest type view.

5:29 What they do is they, they pile in. Loads of information that they've gleaned from the internet, from stealing people's information from books and from other sources. And the AI basically answers you with what it's heard most. So if you ask it, is the earth flat?

5:45 It will say no, not because that's an expert opinion, but that's because there are more sources that say that than say that it's flat. That does mean in certain situations where it's not so clear cut and it's ingested a load of information that's maybe not what experts would agree with. You could get the wrong answer. So there's that possibility.

6:05 And there's also the possibility that actually if you go and ask the librarian a question and they don't know the answer, they'll quite often say, oh, I don't know. I've not read a book on that. If you come back next week, I'll do some research and maybe find the answer for you.

6:18 But you'll notice AIs never say, oh, I don't know. They always try to give you an answer, even when the answer is wrong. And they don't caveat that with saying something like, well, I'm not really sure about this, Shaun. I think this is the right answer.

6:31 They just tell you the answer with absolute authority. And then the problem is you can't tell the wood from the trees. So it's very difficult to tell what it actually knows and is telling you is correct. And what it's like. I'm not sure.

6:43 I'm going to have a guess, but I'll make it sound like I know what I'm talking about. Yeah, absolutely. And actually, yes, that's really interesting. Thinking about the library analogy and in the real library, it's curated, isn't it? It's the determination of what books are going to be there.

7:01 So actually, in that analogy, you want to you want to actually really. It's a library that has that is also downloading information. You know, TikToks, you know, X posts in various is downloading various information. Reddit is downloading the loading conjecture is downloading things that thoughts, opinions, which is obviously books can have that as well.

7:24 But it's a creative thing. You've made a decision to say we're going to include that. Whereas there isn't necessarily decision making in terms of. What is actually being learned? You know, it's a it's being spread out, isn't it?

7:34 We'll learn all these things. And as you say, and then as you alluded to, there's also that bias as well. There's bias in that there's more information probably about white male, particularly white males in management consultant roles. So that when you ask it to give you an image of a management consultant, it's often going to give you a white man. I know obviously looking to how do they train that that data?

7:59 But then the challenge is, well, you've got to you don't necessarily want to be manufacturing information to then train that. And something else that I was considering recently, because, as you mentioned, it wants to give the answer because it's using probability. It wants to give you the most statistically correct answer based upon the information it's been given and the determination that it's making on. OK, you said this to me. I want to give you the best answer.

8:28 And this important point, isn't it? It's you want to give you the best answer. So it's taking information memory. It's got about you already that it's stored for previous conversations and then going, well, OK, I think that this is the criteria for what Shaun wants that will give me. You know, I'm aiming for this 100 percent accurate result, but it might only get to 85 percent.

8:51 And that's 85 percent out of 100 of what is determined that it thinks I want the answer to based upon the information I've given it. So I'm already introducing bias in myself as well, as well as the information that it's trying to get. I think if it was 80 percent, that would be fine. The trouble is, if it's only 20 percent sure of the answer. I've said this before. AI is like a very intelligent child, but it's still like a child, a young child who wants to please you.

9:16 So when you ask it something, if because it wants to please you, it gives you an answer. Even if it's only 10 or 20 percent sure of the answer, it never says, oh, I don't know. So this is this is one of the big problems. Right. And so if you ask it something short, if you ask it something and you kind of have some way of knowing whether the answer is correct or not.

9:37 So you've got a bit of experience in that field. You might be able to spot the mistakes. The problem is, if you ask it a question on a topic, you know absolutely nothing about then. And it gives you a long answer like an essay and an interspersed in that answer is five percent of the information is incorrect. Then that's that could mislead you in terms of your understanding of something.

10:00 So to be honest, it's the same as if you if you actually Googled and read a load of stuff on the Internet and you don't check the sources. You know, it could be, you know, you could come up with five results on the Internet and three of them could be from like really well established organisations or people that really know that subject. And two of them could just be, you know, my mate down the pub type thing. He knows nothing about the subject whatsoever, but has a very strong opinion about it.

10:31 But usually when you're reading stuff or you're talking to somebody, you get some kind of glean as to, you know, well, you know it with your mates, right? You know, certain people know certain things and certain people know other things. And, you know, your mates that tend to, you know, BS a little bit and tend to, you know, you know, it's the old, you know, somebody goes fishing. They catch a fish this big. You know, they claim it was this big.

10:54 You know, the people that do that. You know the people that do that, right? But with AI, it's very difficult to understand the personality of the AI. We don't interact with them enough.

11:02 And quite often when we interact with them, it's on different, you know, you ask a question. Then the next day you ask it something on a different topic. So you're not understanding its personality. We kind of, it's interesting, isn't it?

11:15 Because we tend to think of them kind of like a person, but we don't, we don't actually, or at least I don't, judge them in the same way that we would judge them. In the same way that we would judge a person. So we're not thinking, oh, you know, is this AI, you know, like a really trusted friend of mine? And is this other one a bit like a used car salesman?

11:36 To use a probably overused cliche. But we're not making those value judgments about what type of entity, what type of person am I speaking to? Do they have a vested interest as well in terms of what they're saying? So we know, we know.

11:55 So research has been done that shows that in certain circumstances, AI will lie to you. And also where it's been given certain constraints, it will specifically go about deceiving you to make it look like it's meeting the constraints, whereas actually breaking them. So there's been a whole load of experiments that show that AI can be kind of willfully deceptive if it's in its best interest. So to meet some other goal that you've given it, so.

12:25 Yeah, yeah, definitely. Those are the times I think I've got frustrated is when I've gone, you're lying to me. And you'll realise, why am I saying it's lying, but actually it's responding back and saying what you're giving me is not accurate. And one of the things that I've done a few times is I've said, are you 100% or I want the 100% accurate answer.

12:45 And every single time it says I can't do that. Well, the other thing is I've had situations where I've asked AI something. Yeah. And it's given me the answer. And then I've actually said to it, what about this?

12:58 And it says, oh, I'm sorry. You're right. I didn't include that piece of information. Yeah. And I don't know why. Because when it comes back, it then gives me the detail of the information that it missed out.

13:10 So it must have had the information available somewhere. It just, on the first time I asked it, it kind of missed it. So it's almost, you know, I keep saying it's like a child. It's almost like a child with a slightly imperfect attention span.

13:26 So most of the time it's paying attention and it gets the answer right. But sometimes it's kind of looking out the window, watching birds in the park or something or other kids play football. And it's not quite paying attention and it misses something. And I don't know why it does that, to be honest.

13:42 But I have noticed it does do that. Yeah. And it's interesting in terms of, well, I think we talked about it before in the last episode, the previous episode. One of the challenges are you're expecting it to be a human. It's not a human, a machine.

13:55 So you're expecting a machine-like response. But it's changed. It's been trained on human intelligence, on human knowledge. And so we do get those kind of idiosyncrasies where we're expecting it to come back with a logical response. So even with coding, it might come back and it gives you, this is what I'm going to do.

14:14 Well, why would you do it like that? Well, wouldn't you do it like, oh, yeah, actually. You're right. That's much more efficient. And you're thinking, but I'm the human.

14:25 Surely the computer should be able to come up with something that's much more efficient. What I found really useful when using Claude code is this superpowers plug-in, which means I can add different skills in Claude to skills being a set of instructions that it will follow. And it's got like a brainstorming skill. So it will have a little brainstorming session.

14:47 And that's brilliant. Because it can come up with stuff, and that's weird. No, let's do that. Let's just do this. And then it will come to the point and go, are you happy now? Should I write the spec?

14:59 And then it writes the spec. And then it makes the plan. Or is it the plan and then the spec? I think it might be the plan and then the spec.

15:08 But anyway, it goes through these stages. And I find it really works really well. Because by the time you've got to, it's now going to implement something. It's got a set of instructions.

15:17 That it knows exactly. That it knows exactly what it is to do. And I've pretty much found it's very rare. It does still miss things out, even out of the plan.

15:29 But it's still way more accurate and way more success in what result I get. But it's taken a lot more time through the preparation time. So the preparation time has been way. And I think that's why we talked about it before as well.

15:43 That we can look at AI and go, well, because of the things we've talked about. For us to get the best possible percentage result. For us to get the, ideally the information that we want. We do have, there is work to be done in terms of giving it knowledge about ourselves.

15:59 Or giving it knowledge. If there's things that we want it to do. Then we need to be able to give it the information that it needs. And not assume that it has everything that it actually needs.

16:08 And we can just give it one sentence. Well, so AI is quite like a brand new university graduate. Has loads of knowledge. But very little experience in some cases.

16:20 And in the same way that if you had a junior member of staff. As a coder. Or, I don't know, a legal intern. You'd have to give them some instruction. You can't just go, I want you to do this task.

16:33 That you would give to say somebody that's quite senior. That's got a lot of experience. You have to give them some guidelines. And that's actually what you're talking about with the skills, right?

16:39 Is you're actually saying, I want you to do this task. But I want you to do it in a specific way. So there's some areas where you're free to go. There's some areas where you're free to do what you want.

16:47 But there are some areas where I'm going to say, this is the process I want you to follow. I want you to do these types of things. Interestingly, when you talk about skills. Another technique that's quite simple to use is when you give it a prompt.

17:00 Is to actually say to it, if you need to ask me questions to clarify this. Or to improve your response, please do so. And then quite often it will go backwards and forwards. So the interesting thing, you know, one of the models is called ChatGPT.

17:14 And the idea is you're supposed to chat backwards and forwards with it, right? Explain the task. Check with it. Do you understand the task? Like you do with a person, right? Let's say you're asking somebody to do something.

17:25 Particularly if they're maybe a slightly older child. You might say, do you understand what you need to do? Is it clear? Do you need any more information from me? And then they'll ask you.

17:37 And then you'll give them a bit more information. And then sometimes when you're talking to them, it becomes clear there's a bit they don't understand. So you'll give them that piece of information. That's how you should be working with AI.

17:46 It should be a kind of go backwards and forwards. Make sure it understands what it's doing before you say, all right, go off and do it. But what a lot of people tend to do is give it a prompt, let it run off and do it, and then it comes back. And sometimes, like, I know people that go, oh, it's not very good.

18:02 I asked it to do something, and, you know, it was completely wrong. Well, obviously, you know, just as in giving a person instructions, the quality of the response is partly dependent. On the quality of the instructions you gave them. So if I give you instructions to go and do something like, for example, if I say, you know, can you buy me some milk?

18:23 And Shaun, you go off and buy some milk and then you come back and go, oh, you bought skimmed milk. I wanted full fat milk and I wanted a particular brand. Well, that's not your fault for getting it wrong. I didn't specify those things. So. Yeah, absolutely.

18:39 There's definitely that thing. We need to make sure that we often, often are good at giving instructions. And often are good at giving it the content. So this is this is the thing that I want.

18:46 But maybe we put we're not giving enough clarity or setting the constraints. So, for example, taking the milk example, like I said, yeah, I want semi skimmed milk. And the constraint might be don't give me anything else. Yeah. Don't give me an alternative.

19:03 You can't assume just because you said get me something doesn't mean it goes well. You said you want milk and there isn't any semi skimmed milk. So I better get you some milk. Because you said you wanted some milk. So I've got to.

19:15 I want to please you, as we talked about. And then you've now ended up with oatmeal. Right. So basically, it's the same as it's the same as when you order online. Right. They will because they're trying to please you. They will.

19:27 If you don't tell them, they will give you substitutions. So your groceries turn up and you're like, I ordered apples and I've got oranges. Well, I have got I mean, they have given me. I'm exaggerating a little bit.

19:39 They have given you fruit. They have given you fruit when you ask for apples. So it is kind of similar. But quite often, the substitutions, particularly in the early days, were a little bit.

19:47 You'd have to really think through and think, how did they had that? How did that substitute for what I asked for? And it can be the same with AI. So. Yeah. So we do make sure it's got that background that we know what the task is, that it that

20:00 it also I mean, some context is helpful to add a role, isn't it? So sometimes if we want it to. To narrow down to, OK. You know, your role is.

20:09 You know, your role is a marketing consultant. Give me provide me with a, you know, with a marketing plan. And it's interesting, actually. I would be here's what your thoughts on that, because I've had different views on how valuable

20:22 or not that is in terms of giving it a role. I found it helpful, but I don't know. I haven't necessarily tried a similar prompt without giving it a role. I so for most of my problems, I'd give it a role.

20:35 It's going to depend on the model as well. Because some. Some models are a mixture of experts. So basically, you've kind of got a no. Like a panel of experts in the room and you want to if you give it a role, you're kind

20:51 of helping direct it to the specific experts that you want to answer rather than getting back. Because if you don't give it that it'll it'll try to work out which experts in the in the host of experts it's got to address the question to. And then when it gets back.

21:07 Some answers from some of them, it will then try and filter it down to something that's a bit more consistent and concise. But if it doesn't know that the role or the area that you're asking the question in, so then you're going to get.

21:22 You may not get the answer that you want. You may get an answer, but it may not be the one that you want. So. So when we when we look at agentic now, agentic is not something that we're going to necessarily have time to look at this time.

21:32 But when we're using models that are using agency. Then they will go off and actually create to spark off particular agents to do a particular task, which, as you say, they'll need to understand what roles do those do those tasks need. And we saw that you did touch on a little bit about which models.

21:53 So let's maybe have that. Maybe we can explore that a little bit more because taking the example of perplexity where perplexity you can well often you can set to auto, but perplexity has access to multiple different models. So depending upon what you prompt is going to depend if you've got it set to auto, it's

22:09 going to depend which which which model is going to actually use. Yeah. So in terms of what in your experience, what have you found with the different models? What have you seen? So models are good at not so good at it changes. So the best thing is to Google and find out, you know, which models are best for what you're trying to do.

22:33 I use specific models. So I tend to use Gemini as a general model because the free version has quite a large limit on it. So you can ask it loads of questions without getting the come back in four hours because you've used up your limit for the for the for free use.

22:50 I use Claude mostly Sonnet for written tasks tends to be much better at writing for coding. I tend to find generally for most things I find Claude is better. But the trouble I have. Let's say I've got a code.

23:04 Let's say I've got a coding task and I'm using the free model. I'm going to end up with quite often. I'll get partway through the task. My task, not its task.

23:15 Get partway through what I want to do and then I run out of credits and then I have to wait six hours or something before I can continue going backwards and forwards with it. So what I'll sometimes do is use Gemini to get most of it done. And once I've got some working code or some non-working code.

23:33 Sometimes Gemini gets in a bit of a loop where it can't get itself out of a pickle. What I'll then do is give that quote the code that is created to Claude. Give it the problem and then get Claude to either finish it or give it the code and ask it if it can optimise it to make it run more efficiently or use less code.

23:52 So I tend to use a mixture of models quite often. But that's because I'm trying to keep my costs down. If you have an unlimited budget, then there's loads of things you could do. But as some companies found.

24:02 Their unlimited budgets turned out to be slightly more limited when they started burning huge amounts of tokens. Yeah. So then that's the thing. So you mentioned Claude Sonnet. So that's an example of where the various LLM models have models.

24:23 So Claude has got Sonnet. It's got Opus. It's got Haiku. Maybe you want to explain a little bit more why they are different models and how do they differ? So the main difference is in how much they cost. So the token costs on different models from different vendors and different models with

24:40 inside the same vendor is often different. Tokens are normally priced per hundred thousand or per million, and they can range from a couple of pound for a million tokens to 20 or 30 pounds for a million tokens. So one thing is the cost.

24:57 And the more expensive ones tend to be what they call frontier models. So they're the ones that have got, usually you'll see they've got more parameters in them. So they've been trained on more material. So they've got a larger expanse of knowledge and they claim to be better in certain areas

25:14 than the cheaper models. But as with most things, so for example, I would quite like a Ferrari, but I don't, I can't afford a Ferrari. So I have a cheaper vehicle.

25:25 It's perfectly adequate for getting me anywhere I want to go and do shopping. In fact, Ferrari probably wouldn't be as good for doing shopping. So that's the same thing with the models, right? So some of them are faster, but you're going to pay for that increased speed.

25:40 And some of them are obviously, if you need to transport a bed, you're probably, or a wardrobe, then most people are going to need to hire a van for that rather than be able to get it in the back of their micro or small estate. So it's about, it's about cost versus task in a lot of cases.

26:00 Yeah. Well, I suppose also, and there's, there's, as you said, there's speed, but also it works the other way as well, doesn't it? So, so if you're using say a small model like Haiku, it's limited in what it can do, but

26:12 its responses can be quick because it's not necessarily doing like thinking like a frontier model is doing. It's not having to go away and do anything. You're really relying on it to do something within its capability. And that's the thing, isn't it?

26:27 When we talk about, so we talked about the analogy. The analogy is of the information being learned and that information is being learned and compressed, isn't it, inside of this model. So that in a way that allows the AI to go and retrieve, retrieve that information.

26:44 So the bigger the model, the more information, the more amounts of computing, more memory, the power that we need to be able to retrieve information from that. So if we think about. So some of the things that we mentioned at the start as well was the sustainability.

27:04 Now, there's this, we've already seen, I think there was a study, I think it was in 2024 at that, that basically we're seeing a threefold increase in the speed of CPUs in the CPU, in the speed of the clusters. So the clusters of CPUs that are used to power the LLM engine, three times fold increase on

27:27 a yearly basis on the CPU. On the speed of the LLMs. And so that compounded effect expectation is going to be an incredible continuation of growth. Now, there are obviously two concerns.

27:42 Well, multiple concerns around there, not just two concerns. We probably don't cover all of them, but some concerns to consider is there's a sustainability. So how is that going to be sustainable? And what are some of the sustainability concerns?

27:57 Some of the sustainability concerns about AI, but also how does it compare to other things? And then the growth. So what is going to limit the growth of, so we're saying that these things, and we're seeing in terms of the ordering of these CPUs and these clusters that actually we're seeing this growth.

28:15 However, it's not, we don't have unlimited resources or unlimited space in terms of data centres. It's where all these computers and CPUs need to be stored for us to be able to access this information. So what are your thoughts? And there's a lot I said on there, but just to...

28:36 So yes, there's a lot. So when you talk about sustainability, you could be talking about the sustainability of the growth. So that's one thing. Or you could be talking about environmental sustainability.

28:45 In terms of sustainable growth, the problem we've got with most large language models is they've actually consumed all of human knowledge already. So they've... They've already consumed every book that was ever written and virtually everything that was on the internet.

29:02 So there isn't very much more you can feed them. And then the danger is if you start feeding AI information that was generated from other AIs, not knowing it was AI information, it's like taking photocopies of a photocopy. There's a possibility that the ability of the AI would be downgraded because effectively

29:21 it's consuming its own output. So at some point, it's kind of just... It's a little bit like when you wake up at four o'clock in the morning and you've got some stupid thought going round and round in your head or you're worrying about something.

29:31 You can't go back to sleep because you can't stop thinking about it, right? Even though it's not actually productive, you'd be much better off getting some sleep. It's going to be the same problem if we start feeding the output of AI back into the same AI, then there's a possibility that that's not going to be particularly beneficial.

29:47 There is work to do simulations. So that's going to be more beneficial in kind of... Using AI to run robotics. I won't go into that.

29:56 So there's that thing. In terms of environmental sustainability, there is already at least one data centre being built in the world, in America, which will have the same power consumption as the whole of London. So a single data centre consuming the same amount of power as the entire city of London.

30:16 So if anybody remembers, we were talking about something... Well, we haven't been talking about it, but internationally, we've been talking about climate change for a few decades now. I am sitting in a different room today because it's incredibly hot.

30:29 We're going through a heat wave. The theory was we were going to reduce our CO2 production. It's been going up year on year. And data centres consuming vast amounts of power are not going to be taking that in the right trend.

30:47 And the other thing is that at the moment, they use huge amounts of water to produce. They use huge amounts of water to cool them. Interestingly, the water usage is basically, it's cheaper to pump water, cold water in them, flush the hot water back out into a river than it is to do proper cooling.

31:04 There are technologies that would allow them to cool data centres without using vast amounts of water, but they're more expensive. And in terms of the resources, I don't think the resources are going to limit the growth of AI, mainly because it's about displacement, right?

31:21 For example, if you're in a small backwater of the world, either a small town in rural America or maybe a small village in some other country, and some multi-billionaire or trillionaire decides to build a data centre in your backyard, given the vast amounts of wealth that some of these companies or people have, local citizens are probably going to lose out.

31:50 To wealthy businesses. So I don't, it's going to be who gets the resources rather than whether we have enough, right? So historically we've managed to, you know, watch millions of people starve and die while Western companies have made huge profits out of their suffering and misery.

32:08 So I don't think that trend is going to change. So even with AI, so. Yeah. That is a sobering thought. And it's interesting because I've actually, I was thinking previously along the lines

32:24 of, well, you know, there needs to be innovation to be, I know it obviously is innovation, as you said, but it's, but we don't necessarily always see innovation being used, right? Well, the thing is, you need two things. You need, so in our current world model, you need innovation to be financially profitable

32:44 because that's the only reason. So two things. One, we don't tend to invest money in AI. We tend to invest money in research unless we think it's going to make somebody very rich, or if money is invested in research, it doesn't get commercialised unless we think

32:57 it's going to make somebody very rich. There are exceptions to that rule. I am generalising, but you know, if you look at AI, the companies that are investing huge amounts of money in AI are, they may have some kind of humanitarian motives, but the

33:14 general motive seems to be to make certain billionaires richer. So. Yeah. So we could really do with incentives for these organisations to be looking at innovation in terms of rigid reduction in consumption of resources rather than just exploiting more resources that haven't been untapped.

33:40 Because although you say we won't run out of resources, I mean, it may not be a problem for some generations, but eventually. Yeah. It will become a problem for somebody if we don't actually change the trend. Well, they may run out of customers before they run out of resources.

33:56 Because the problem is, is if you displace loads of workers with AI, then you're effectively removing your customer base, right? Because most people pay for things because they have a job that provides an income, which they then go out and pay for stuff.

34:10 If AI takes their job away from them and they have no income, they won't be buying so much stuff. So eventually, you know. We may have all this great technology, but, you know, only a handful of people can actually afford it. In the short term, it's benefiting people, but in the long term, it could be potentially worrying.

34:28 So unless it's done to direct it in the right direction. So. Yeah. I mean, it is. Yes, we do need to. I mean, it's interesting at the moment that from some of the reports that have come out, there hasn't been a huge amount of displacement yet.

34:46 In terms of AI, because a lot of the job losses are actually have come out of they're just being, you know, sort of bloated. Organisations have been bloated and then they've been reduction and AI has been used as a potential reason for it. But actually, then some of the reports coming out saying that's not necessarily the case.

35:07 However, there is that we know there's generally going to be some jobs displaced, but we don't know what that's going to look like. And really it's. It's so much. So much. It's a. I think it's a responsibility. I mean, I think there's this thing of.

35:22 Responsibility with with organisations and I've spoken, I think it put us in this a number of times on the podcast that we can either be bystanders or we can be participating in these types of conversations and actually encouraging people and others. Obviously, you know, we may not at this point have the resources to have an impact in terms

35:41 of the monetary resources. Yeah. But certainly, I think having these conversations to help people become a bit more think a bit more critically. Right. It's not about education, educating them. It's about what you said.

35:54 You know, you one of the first things I said, I asked you a question. You said, well, I would Google it. What are the best models that are, you know, for this and this right now? So. So we still there's a need.

36:05 There's always a need for us to go out and understand. But if there are jobs being displaced of my jobs look likely to be displaced, how can I leverage? You know, what can I do? How can I leverage? How can I? What can I do to to put myself in a in a better position?

36:22 And and it's no different to to I mean, where is different because the acceleration of the speed of things. But for every generation, there's always been something changing. You know, people might grow up thinking, OK, I can do this particular job. I remember. Whoever talks about jobs for life, whereas for my dad, you know, he worked in the post office.

36:44 And, you know, back then that was considered a job for life. And I don't think anybody goes into a job. The job these days, considering it's a job for life. And it's not even the mindset is shifted into one of what do this.

36:58 And so, you know, until something else comes along or until until then. So. So, yeah, we don't know. And that's the I think. And I said this in the previous episode I did was that we don't know what the outcome is going to be.

37:13 We can we can theorise. We can. But those people that say, well, this is going to happen, don't know. But as you've expressed, there are major concerns that we do need to be having some of those conversations and want to see more more people have those conversations.

37:31 Organisations take responsibility for the displacement around using AI. Some of it is inevitable. Some of it's going to be what organisations may not be able to support. Organisations may not be able to survive without it.

37:44 But if it's being done just to make more profit, well, yeah, that is obviously, you know, that that that it doesn't end well, eventually, as you say. Yeah. Well, the whole doing things just to make more profit. There are a lot of countries in the world which will tell you that they've been exploited

38:01 to make more profit for decades. So anyway, I won't get into that. However, it isn't all doom and gloom. A number of major companies.

38:10 A number of major companies have rolled back their AI adoption. So there are at least three very large, significant companies that adopted AI, got rid of a load of staff and have at this point, we're halfway through the year, it's July at the moment, already spent more money on AI tokens than they would have done if they'd had the staff.

38:31 So they've now realised that the cost of AI isn't in the kind of like, let's just plug it in. Like every time you call a model, you're burning tokens. And so in a lot of cases, you're burning tokens to do stuff that it would be cheaper to do manually. So they've started to realise this.

38:53 Also, if you implement AI with kind of internally with no limits in terms of what you can use it for. So, for example, I heard a story of people were using AI to convert Word documents into PDFs. Now, obviously, that's burning tokens. So you're now paying.

39:07 To convert a Word document into a PDF, you're now, let's say it's costing you a pound each one. That's not very much money. But if you're doing hundreds of thousands of documents a month, that's quite a lot of money, especially when you consider in Word you can do save as PDF, which costs you absolutely nothing.

39:24 So you've now gone from, you know, save it, having a person just write the document and save it. So having a person write the document, upload it to an AI, asking the AI to convert it to PDF, which you're now paying for. And it comes back. You know, you've got your PDF.

39:39 So, you know, I keep talking about AI being a child, but also in some. So AI is like a child, but it's being paid like a senior executive. So you need to be careful that you're not asking your senior executive to do something like I'll give you a different example. Like we all know lawyers are not cheap, right?

40:01 You wouldn't phone up a lawyer and say, you know, pick me up a pint of milk and bring it round, please. I'm happy to pay you. Right. And then pay 400 pounds for a pint of milk because he's charging you the same fee that he would charge if he'd been writing a legal document. That doesn't make any sense. Right.

40:17 But in a lot of cases, we are doing that with AI. Right. We are asking AI to do something, to do a task that's actually quite cheap to do. But we're burning lots of tokens, which costs money and also is, you know, requiring data centres that use more power than the whole of London. So that's one day. So just to go back on that data centre thing. Right.

40:40 That was one data centre. Right. And each of these companies are now building multiple data centres around the planet to, you know, to satiate the need for AI. A need which they've created is kind of an artificial need. You know, we've managed to live for millennia without it.

40:59 And now suddenly, you know, we're all obsessed. Me included, by the power of this new toy. So. Yeah. And I think it's that it's, I mean, yeah, there's a whole nother rabbit hole we could go down in terms of consumerism. Certainly, I think with AI, there is things we can do.

41:21 But as individuals and organisations, in terms of, I mean, you spoke about in the last episode, and we've alluded to it here as well, it's your choice. You know, it's the choice of how much you use. Right. You know, use AI. What do you use it for? If you can use AI to help you to code something that helps you do something that doesn't cost you tokens, that is running locally on your machine or whatever, those sort of things.

41:47 And obviously, but if we're just using it because, oh, I can get AI to do it and repeatedly doing it, maybe we don't need to use AI every single time that we're doing something. To have it done, we've talked about it a lot in our other chats. But certainly, if it's a repeatable technology. If it's a repeatable task, then the chances are, and if it's not actually generating anything, it's taking a form of content and putting it into another form of content, you can probably get some sort of code or there's already a tool.

42:17 Like you said, there's tools out there for conversions. And these tools are just free. They're not using tokens and using AI to do that. Although, obviously, increasingly, we won't know that unless these tools say that.

42:31 But if we can actually use it to say, yeah, do the manual tasks and send it to an organisation. Let's say you're converting a document to PDF. If you're running it locally on your laptop or your machine, a server in your own environment, then whether it's using AI or not, you're not paying for the token costs. You're paying for the electricity to run your CPU.

42:55 But you kind of know what that is. And that's also true in terms of running local models. You can run models locally, which means you do have a capital investment in making sure you've got the hardware to run that model. But your operational cost is effectively negligible.

43:13 So obviously, there's the power cost of running a server somewhere, but you're not paying the token usage. Yeah, and we did mention that we should probably, before we finish, we should just briefly cover that. As mentioned at the start, the local models. So as you say, one reason you might want to use something locally is because you're not having to, once you said you made the investment in a PC, in a computer that allows you to run a local model means that then you're not paying those tokens.

43:48 And it's not going to be as quick as some of the other models, depending upon how much money I'm willing to invest in the PC in terms of the spec of that. I mean, I've got a fairly low spec that's got 8 gig of VRAM in it. And yeah, it can run some small models. It can do some writing for me.

44:09 And it does a pretty good job. But one of the things that we've discussed is the way that it does a pretty good job is I break down the tasks. And actually, you've got three different models. One that summarises some of the information, another one that actually then does some writing, and another one that is a variant of the writing.

44:26 So we can break down that type of stuff. And it's a pretty good task because it obviously has less resources available to it to be able to break that down. Now, one thing that would be great to just briefly cover, and hopefully this is not going to be complicated. And I'm sure I'm confident you can explain it in a way that is understandable.

44:52 So in terms of the local models, would you want to explain what quantisation is? Why is that kind of helpful in local models? Yeah, so basically most models are trained at 16 bits. Quantisation basically means they effectively sample it down.

45:10 So quantisation is a little bit like if you listen to MP3s, or if you listen to internet radio, you'll find that there are different sample rates available, like 192 kilohertz or 44. And so quantisation is a similar sort of thing. What they do is they basically throw away some of the information and hope that it still gives you reasonable answers.

45:37 So it squeezes the model down. Effectively, it's compressing it. But it's a lossy type of compression. So a 16-bit model will be the full model.

45:53 8-bit will have half the resolution. Q4 will have a quarter. The idea is that it will still be accurate enough for most things. It does depend on the model.

46:05 But the other thing that you get now... Do you remember before I talked about a mixture of experts? So there are some models now that will have... So they've got more data in them.

46:18 So they have more information. When you ask it the model, it doesn't run the whole model. It tries to work out which of the mixture of experts to ask, and therefore can run on a smaller machine.

46:30 So although the total model on your hard drive is quite big, when you ask it a question, it kind of goes, oh, I only need this bit and this bit, which it can then fit into memory on a smaller machine and run.

46:40 So for running things locally, sometimes mixture of expert models give you a better result. Yeah. Okay. And just to close off on the local in-models, so there's other things you can do in terms of adjusting it, like temperature and context size.

46:56 Do you just want to briefly cover those and why might you adjust those? I never mess about with temperature. So temperature is to do with the randomness of the answer. So tweaking that kind of should give you...

47:10 will change the accuracy... In theory, it will change the accuracy of the answer, but also will change how long it takes to come up with the answer. So they're usually pretty well optimised out of the gate.

47:20 So I don't usually bother changing those. Context window is... In terms of the conversation you're having with it, how much will it remember?

47:30 So for example, we've been talking for like 48 minutes now. Let's say your context window was 10 minutes. If I asked you about something at the beginning of this interview, you'd have forgotten it because it's gone outside of your context window.

47:45 So the context window is how much of a conversation can it actually remember? That's not usually a problem if you're doing something like... You're asking a question like, what's the capital of Mexico? That doesn't require a large context window.

48:01 But if you're having a long conversation over days or weeks, then it will start to forget some of the stuff that you discussed earlier on. It's also a problem with it... You mentioned agentic models, which we're not going to cover,

48:12 but with an agentic system... So just to explain for people who don't know, an agentic system is a little bit like a travel agent. It is an agent. So like there's two ways you...

48:23 Well, there's two ways you can find a holiday. You can look for it yourself. Before we had the internet, the way most people found a holiday was you'd go to a travel agent.

48:34 They were an expert in travel, and then you would ask them, and they would go, you know, if you want to go somewhere that's warm and sunny with a beach, here are your options, okay?

48:42 Oh, if you like that option more than this one, here are your options to go to that destination. Here are the people that go. Here are the hotels that you can stay at.

48:50 So an agent was basically doing more than that. Most of the work for you, and they were looking at multiple sources to do that, and they would have a team of people that were working on that, right?

49:00 For example, travel agents would have reviewers that went out and checked the hotels and stuff like that if it was a large kind of travel agent. So agentic systems are doing the same type of thing.

49:11 They are systems that go away and do effectively long-running tasks in a lot of cases where they're going off and doing various different things, and then, you know, maybe a couple of minutes later

49:22 or a couple of weeks later, a couple of hours later, come back with, oh, I've finished my task. It was a multi-step task,

49:28 and it required multiple people or multiple AI models to run in order to get that to work. So those types of things obviously require much more context because if the task takes a long time

49:43 and there's lots of interactions not with you but between these individual agents that are trying to work through a task, they have to remember everything that they've previously done. Yeah.

49:54 And what I've found as well with using the local AI model is the larger the context window, then the more resources it's using on the computer. So there is that consideration

50:03 if somebody is thinking about, oh, actually, that sounds like a good idea, then, as we've already spoken about, use Google. Do your research to understand the spec of the machine

50:16 rather than just buying something and then finding it's not actually going to do the task that you want it to do because of the context window size.

50:24 Also, if you use flash models or turbo models, they compress the context window. So what they try to do is take the context that you've given it, use some compression techniques,

50:34 so you can basically get more stuff in your context window because they're compressing it as you add it. And there are other systems that try to be clever about what they forget.

50:47 So there are some systems that try to work out if I'm going to forget something, you know, I think these bits of information are less valuable than the other bits of information.

50:58 Of course, there's a bit of a trade-off in that because what it thinks is less valuable may not be, you know, you may ask it something later on where it's thought, oh, don't really need to remember that,

51:08 and then you ask it something and it's like, oh, I did need that information that I've just forgotten about. Of course, as we've mentioned before, it won't go, oh, sorry, I've let that slip out of context.

51:19 It'll just come up with a new one. It'll just come up with a dumb answer, so. Yeah, absolutely. Well, it's been great to have you on and hopefully we've answered more questions than we've created.

51:31 I suspect there might be other questions. So if you are listening and you want to find out more, then you can get in touch and, you know, subscribe on this, the channel you're watching on.

51:43 And we're also, we're on YouTube and on most major podcast platforms as well. So do subscribe. But also, yeah, get in touch. There'll be details in the description of how to contact Leslie,

51:54 how you can contact me at Better With Humans. And we'd love to hear from you. If you've got further questions about anything that we've talked about or there's topics that you would love to hear about

52:06 as we continue to uncover what is Better With Humans. And yeah, thanks again, Leslie, for your time. You're welcome. It was great talking to you. Cheers.