Ransomware in the AI Era: Evolution, Hidden Weaknesses, and Incident Response with Behnaz Karimi

TLDR;

• When it comes to AI Ransomware Attacks, there are many entry points like models, ML Libraries, Pipelines, Datasets, Prompts, and others. Each area should be paid equal attention to secure the AI systems.
• For securing AI systems, organizations should focus on reachability to AI systems, review each component, such as Supply Chain, Pipelines, ML Libraries, etc. Do an honest review of each and prioritize the least secure areas.
• For security in general, take a moment to analyze your AI systems for tools & use cases, risks associated with them, chance of occurrence, Impact of the risk, and set basic controls & governance around it.

Transcript

Host:: Hi everyone, I'm Purusottam, and thanks for tuning into ScaleToZero podcast. Today's episode is with Behnaz Karimi. She's a cybersecurity engineer and independent researcher. She focuses a lot on ransomware and agentic systems. She's founder of Tramarena and co-leader of OWASP AI exchange. And she specializes in AI security with over 14 years of global experience in cybersecurity.

Thank you so much Behnaz for joining today with me.

Behnaz Karimi: Yeah, thank you so much, Puru, for the introduction and a hi everyone and thank you so much for your invitation. I'm really happy to be here today.

Host: Same here. Before we dive into some of the security areas, we generally ask this question to all of our guests and we get very unique answers from this. So what does a day in your life look like?

Behnaz Karimi: So my daily life, it's looked like that every day I'm trying to update myself actually about new technology and about everything. so basically, as you know that I'm founder of Termarina. So my life is divided into two different dimensions, I would say, in neuroscience and cybersecurity.

So every day I try to find out a new technology and search about the I would say new research as regarding to the ransomware attack because I would say this is very important topic and that would be actually more complex in future.

And on the other hand, one of my part of the job as a colleague in OWASP AI exchange is working and talking with different people from the founders, co-founders, and investors, and security engineers really enjoyed it. So basically this is the whole of the things that's happening right now in my life.

Host: Awesome. So on one hand, you are learning new things and how to, what are new ransomware attacks? How do, how does the world stay safe? And then you are working with founders and engineers to sort of share that knowledge so that they can utilize that information and build secure systems. That's amazing. That's amazing!

And that aligns really well with the topic that we have today, right? Which is like securing AI systems against ransomware.

Behnaz Karimi: Exactly. Yep.

Host: Can we use threat modeling or what are some of the real world controls? So let's dive in. So ransomware has been one of the prevalent attacks to organizations since decades. In 2026, I'm sure ransomware is no longer just about encrypting files on a server. What have you seen? Like what is one key element that you have seen which has changed? Let's say today, we're comparing it with let's say 10 years ago or five years ago.

Behnaz Karimi: Okay, yeah, absolutely you're right about the change. So the traditional idea of the ransomware was, you know, the locking files and demanding Bitcoin. That's really becoming outdated right now. If you look at the 2025 and 2026, ransomware has evolved quite a bit. It's not much more about multi-layered extortion more advanced tactics, often even AI drive and runs over.

And there is a clear shift towards simply stealing data instead of encrypting. So I would say in fact by 2026, a lot of groups don't even rely on encryption anymore. Instead of shutting down a system, they steal sensitive data and then apply pressure.

on the other ways, for example, legal pressure, compliance issues, reputation of damage, all by the company's technical is still operating, right? So even if you have a strong backups, you are not really safe.

And I don't want to go to that topic regarding to the insurance issues and all of the things that are happening after that, that an attack happens and incident response right now. But on the top of that, I think is are getting more aggressive.

If you search and look at right now what's happening with the ransomware attacks. So it's not just about the systems anymore right now. They might launch DDoS attacks or even reach out directly to customers, patients or businesses, partners to increase their pressure. Some groups even go as far as threatening to report companies to regulators.

And another important big shift, which is I'm seeing, is what you could call a once too many attack model, right? Instead of targeting one company, attackers go after a managed service provider. And through that, they can hit thousands of the organizations at once.

And if you compare that to, say, five years ago, around 2021, I would say ransomware was much simpler. Files got encrypted, a ransom was demanded, and sometimes you would get your data back after paying. But it wasn't any guarantee that they do not attack again you or they do not sell, for example, your vulnerabilities on the dark web, right? So today encryption, I would say, is almost secondary.

And finally, there is a human factor. That's becoming more important. Ransomware groups are increasingly recording insider attackers, right? And so overall, ransomware today isn't just a technical problem, I would say. It's really becoming a border social and organizational threat. This is what I'm thinking about it.

Host: Wow. So that's quite a shift from. Yeah. I remember reading about ransomware and all five years ago where it was all about that. You see a screen where it says that, your data is all encrypted. If you want this to be decrypted, then you pay us and then we'll do it. But now it looks like with.

Behnaz Karimi: Yep.

Host: There is a lot of focus on data. is a lot of focus. MSP is something that I had never thought about, like going through the MSP providers, because yeah, that gives you access to a lot of customers indirectly in a way, right? Wow, that's a new way. Do you think AI has anything, any role to play here? Like does ransomware affect AI systems as well?

Behnaz Karimi: So basically, answer would be, yeah, answer is definitely. And actually, it goes in both directions. So I would say that AI systems can be affected from ransomware's attack. And we can use or leveraging AI to do a ransomware attack, right? Again, on AI systems or in IT systems, right?

AI actually is becoming a target itself, right? So it's not just about someone getting access. Attackers can manipulate outputs, extract sensitive information, or even inject malicious contact through those systems.

On the other hand, right now, I mentioned, leveraging AI to do attack, AI is also, is becoming a weapon. Attackers can starting to use what we call, I would say agent AI that right now that I'm working on it. Basically the agent AI that we know about it, autonomous system that can make decisions on their own. It depends on the autonomous level, right?

So autonomous attackers can automate and scale their attacks. And we are already seeing early signs of this, right?

For example, the AI-powered ransomware attack prompt like, which uses AI models to dynamically generate malicious script across different operating systems.

So more broadly, ransomware is becoming more autonomous. AI is being used to automate exploitation, analyze the stolen data, and even handle part of the ransom negotiation.

And looking ahead, this is likely to accelerate. And in future, even by, I would say, this year, right, 2026, we could see fully autonomous ransomware pipelines. I would say this is my expectation that we will see it. And meaning even small groups or individual attackers could launch large scale attacks at the level we really haven't seen before.

And if you follow the ransomware news regarding to the new groups that they are coming to the market, you can see they are increasingly, right? And this is something that I would say AI brings on the table for them. So this is, as I mentioned, on the both sides.

Host: Yeah, so it sounds like the more autonomous AI becomes it also becomes a Like it's also a tool that an attacker can use as well, right? They can also build autonomous attack systems Like earlier it used to happen that there were phishing campaigns and they were not that great but with AI you can generate phishing emails in the same tone as your CEO, right?

Based on what you have seen their behavior is and then that sort of helps you that gives credibility to your attack in a way, right? And it relatively it's easier to fall for it for employees in a way. What are like I'm curious now that today what are some of the common patterns you see when it comes to ransomware attacks against AI systems?

Behnaz Karimi: Right.

Host: And I'll come next come to this next that what do you anticipate happen? But yeah, let's start with what common patterns do you see when it comes to an Ansober attacks against AI systems?

Behnaz Karimi: So I would say I can see a different, there are two different patterns, right? And once you see it, you cannot unsee it, right? And that would be the three acts, right? Every time. And I would say the act one, this is a supply thing, always. The attacker,

Host: Okay.

Behnaz Karimi: for John I's model on a public repository. It passes your benchmarks and your team pulls it in. Nobody likes it because it was engineered to stay, dominate during testing. And that's not accidental. That's engineered patience. And the act to persistence, it doesn't seem actually on a server waiting to be granted, right?

It waves itself into your ML pipeline, right? Training is preprocessing hooks, model checkpoints, and it waits. And right now, with the leveraging AI, would say 72 hours specifically or less than, they can wait and long enough to clear your anomaly detection. That's not opportunistic crime.

And that's APT behavior. And I would say the three is detonation and surgical. Every modified gun, every checkpoint encrypted. Inferior endpoints down. And the ransom note, it doesn't hit your inbox and knock, knock. is, this is a ransom.

So it's surface through your own API, um, as the error responses, your own dashboard, the AI delivers its own extortion message. And then finally, the very scary part recovery isn't to restore, uh, isn't actually a restore from backup. It's not going to work like that. It's full pipeline reconstruction, complete retraining from scratch. And I would say days or maybe months, not hours. And every transaction in that window is, I would say, risk. And supply chain infiltration, pipeline persistence, surgical detonation, each stage exploiting something uniquely AI, right?

And traditional security tools, would say, weren't built to see this coming. And this is exactly the gap that right now I'm working on it and try to actually improve and raise awareness about it for the different companies.

Host: Wow, that's like the one thing that stood out from what you said is the persistence, right? That the attackers are not like the moment you download the model, they are not starting the attack, rather they are waiting so that if there are any anomalies detection systems or any systems which is detecting patterns or something like that, it can't detect because there were no activity in let's say 72 hours or something like that.

And you cannot correlate what exactly happened. And that sort of gives you an edge. There are several model repositories. Like you can go to hugging phase and download a model. Or you can go to Olaman and download a model. And if you do not do enough security analysis of the models, that means you will fall prey to these things, these attacks. That's very sophisticated.

I am trying to think in which direction I want to go. Is it because users are not that well prepared or well suited to analyze the models they just take, let's say if there are many like when you go to hugging face, it says how many downloads there has been and you fall prey to that and you download a model which is of attackers or attackers are sophisticated or is it both like.

Why do you think it is happening?

Behnaz Karimi: Both sides. But I want to say that that is the fault of the, for example, the users or engineers or something like that. they were underprepared, right? For example, when you are talking about the hugging face that people go there and download it, they have to be prepared that maybe the model is malicious.

They have to think about it. When you wanted to bring a model on your production, the expectation would be that before bringing the model on production level, you have to test the model, right? At least test on the test environment to find out any malicious behavior.

So the governance frameworks, the integrity monitoring, the supply chain controls for AI artifacts, they are still being written, right? not fault a team or for example that they are for example wrong for a missing controls that I would say didn't exist yet right so but the attackers geniusly they are sophisticated embedding malicious logic that survives your entire validation process right engineered on your testing workflow, a dormant age for 72 hours, that's not a lucky fishing you make, I would say.

That's the APT level patience and precision. And defenders, it's very important that defenders be prepared for that. Right now, defenders, the gap is between defenders and the attackers, right? Attackers, they can do attacks, especially for the ransomware without any boundaries, right? They have the resources and they are not worried about that.

For example, someone tells them that, you are not allowed to do these things, right? But for example, for defenders, the situation is not like that. And this is the gaps between the capability of the defenders and capability of the attackers actually the defenders, should be, they must be like attackers, right?

So at that stage, we can see that, okay, they have the, things that they want. And so the attackers did the attack. So defenders couldn't actually stop the attack. That combination is exactly, I would say why this threat category is so dangerous right now.

And this is the gap and we have to close this gap.

Host: You brought up a very good point now like when it like prior to AI Like this is still an issue though with open source software, right? Like let's say you are writing a Java program you see there is you go to maven repository and you see that there is a package and you just download and start using it and you move it to production right?

And over the years, we have defined processes where, in the CI pipeline itself, we do scanning of dependencies to see what vulnerabilities we are getting and things like that. And we are now sort of seeing something similar in the model world as well, where you should not just download any model. Maybe there should be some, as you mentioned, some testing, or you look at the AI bomb to see what some of the issues are that you might inherit from the model and maybe take action before it goes to production.

Like as you highlighted, you cannot just take any model from, let's say, Hugging Face and start training with it and then go to production without doing any analysis on it, security analysis on it. Before we go to maybe the remediation side of things, one of the questions that a common friend of ours has asked, like Yuvraj has asked,

What kind of system components have to be compromised for a successful AI ransomware attack? Like is it the model? Is it the pipeline? Like what do you think is one of the components that have to be compromised so that an attacker can start their ransomware process in a way?

Behnaz Karimi: Let me keep it this simple, right? Because I don't want to go through to the whole of the details, right? So attackers usually target two layers, right?

The first layer is your AI assets directly, right? That includes your model, your data sets, your pipeline, your prompts, basically everything that make your AI work. If that gets corrupted or locked your AI stops working. And you know that what happens when we are talking about the international companies that they are working with the different countries and they are using agentic. And you know what will happen.

And the second, the underlying IT infrastructure, things like cloud systems, CI-CD pipeline credentials and dependencies. This is the key that attackers often don't go after AI directly, right? And they break in through the normal IT weaknesses, then move into your AI system. And here's the tricky part. The malicious code often stay hidden for a while before doing anything.

And this is one of the points that every time when an incident happens with a ransomware attack, before any restore, you have to be sure that your system is still in a safe mode. What does it mean? It means that attacker may be would be there for example, for six months, attacker is in your network and you didn't realize it, even your firewalls, your antiviruses.

And so before registration, you have to be sure that your model is clean, your model is safe. So by the time you notice the attacker may have already been inside your systems for the days, but you didn't realize it.

So this is the components that will be compromised for successful AI attack, which means that I would say the whole of the AI components from artifacts, datasets, and anything.

Host: I guess there is not a single entry point. There are multiple entry points and you have to sort of secure every single entry point. It's not just about that you secured your model. You are all good. You have to also think about the prompts, the pipeline, the infrastructure, every layer in a way.

So you have worked with large organizations and you have built like massive global digital ecosystems.

What have you seen is the most overlooked entry point? Like out of all of the entry points that you listed, what is mostly ignored out of them?

Behnaz Karimi: So honestly, it's not what most security teams are watching for, right? Everyone's focused on the perimeter firewalls, endpoint detection, instruction systems. And attackers are just walking straight past all of it through the AI supply chain.

And what I actually try to talk about very specifically is about that. That the most dangerous entry point in an AI integrated enterprise is when we are talking about AI system is the model repository, public repositories, third party pre-trained model, ML framework dependencies.

And your team, as I mentioned, that pulls in a model during routine optimization cycle. It passes benchmarks. And it looks legitimate. It performs well. And embedded inside is a malicious logic. And that sits completely dormant during your entire validation process.

And supply chain and third party attacks are right now rapidly increasing. And yet many, I would say, organizations still like even basic AI specific security controls, Like integrity monitoring or proper validation of the model artifacts.

And at the same time, the GenAI prompt in France itself has become a serious risk surface, with frequent cases of sensitive data exposure across organizations. And I would say this is not a niche issue. This is a systematic vulnerability that's happening in plain sight and every single day. And we have to be worried about and we have to be prepared for that, specifically when we are talking about using AI system in healthcare and finance.

Host: Yeah, so we hear about ransomware attacks feels like every single week some major organization gets impacted because how they are using AI systems, how attackers are leveraging. One of the things you mentioned multiple times so far is like the persistence that these AI ransomware attacks are like they just sit silent for days to go undetected and then start the attack.

Now, when it comes to maybe bigger organizations, you might have resources to sort of secure every single entry point and make sure that your pipeline not impacted your prompts, your model, all of that. Do you see a lot of ransomware attacks happening on small or mid-sized organizations as well or like small and mid-sized organizations can say that yeah, we will not get attacked because we don't have maybe enough value for attackers. Like how do you see it?

Behnaz Karimi: No, answer is no. If we, let's consider a simple example about the tradition. I would say the traditional one when I do not mention the origin AI to the ransomware attack.

If you search it and if you can find the information, there are many organizations that they are not famous, they are not rich, but they are affected with a ransomware attack and the attacker demand a cryptocurrency from them. And they were under pressure that their information will be public on the internet. They were not worried about it, but their business was gone.

But think about it right now, we have a small or mid-sized organizations that they are integrated AI on their system. And I wanted to be very direct about that because there is a dangerous myth that smaller organizations are somehow of the broader, right? But they are not, they are the target, right? And because they are starting to integrate AI on their businesses and they have data, right?

Data is most valuable actually, I would say, an asset for any organization. So think about ransomware as a service. It has made it very easy to launch attacks. Even people with little or no technical skills can now rent powerful tools, choose a target, and carry out complex attacks.

The kind of the attack that one targeted big, for example, binds can now hit a small company as well, just as easily. And for small and mid-sized companies, it gets worse with AI.

A large organization at least have security teams and proper controls, right? And they start to invest money on their security controls, but the smaller companies are adopting AI tools just as a fast, for example, like a third-party API, CGPT, co-pilot, without having the same production protection in place.

I would say they often do not track where models come from or, for example, monitor what's entering their system. I would say to reduce the risk, AI models and data should be threaded like critical security assets with proper validation and continuous monitoring. Continuous monitoring is very important because some companies and some engineers think about that the first monitoring would be enough.

No, every single moment is very important. And even large companies struggle with this. Smaller ones usually do not do that. So attackers know this, they know, right? And instead of targeting one company, they can attack a service provider. As I mentioned at the first of talking that manage many companies at once. And that's exactly what happened that right now with many of the organization.

So for your answer directly, no, it's not easier for smaller organization. It's actually more dangerous because attack tool are becoming widely accessible for attacker and much faster than, for example, defensive majors.

Host: There are a couple of things that you mentioned which really like makes a lot of sense. Like one is like you often think that when you are small attackers to not have a lot to gain from you, right? Maybe that's why they will not attack you or something like that. But because you are trying to move fast, you just download any random model, just see how many downloads there were, you just downloaded, you start using it.

You do not do enough due diligence around it and you just embed it into your pipeline, whole pipeline and that stays in the pipeline and then attackers can wake up and start attacking. And there is less visibility for attacker also, right? Like they can go in stealth and they can keep attacking smaller organizations as well.

So thinking that smaller or mid-size organizations are maybe not a focus of attackers is guess is completely wrong. When it comes to AI systems at the end of the day if you are in healthcare even though you are a smaller organization you have access to key data and as you highlighted right that data is the key aspect that attackers ransomware attackers are trying to get to. Maybe they are not focusing on compute or bitcoin anymore it's more around data so that you can sell it in dark web and things like that and make a lot more money.

So, it makes sense. Now, on this line, one of the questions that Yuvraj is asking is how can organizations today understand where they stand against these ransomware attacks? Like how do you do sort of a self-evaluation? If I translate that question, how do I know that where do I stand when it comes to ransomware attacks?

Behnaz Karimi: So I would say first they have to themselves ask two simple questions. I try to keep it simple because the understanding the points is very important. So you have to ask simple questions. For example, can a normal IT attack reach our AI systems, right?

And if an AI component is compromised, can it affect the rest of our IT environment?

And just thinking throughout this will actually relieve gaps most times haven't considered, right? And this is the first question that you have to ask yourself.

And second one is that walk through your system step by step and be honest, right? Think like an attacker. Thinking like an attacker is very important. And ask question, do we only use trusted models? Can models run code in our environment? And if something is compromised, can it be spared? And do we have clean backups?

I mentioned backups many times with the different strategies that they restore from backups and using backups, does, it do not give you this guarantee that you are safe. And if you don't know the answer, that's already a red flag, I would say. And

The third one is that you have to check your visibility. Do you actually know what models and I would say your ML libraries are running in production and where they come from? And if not, I would say that's a big supply chain risk. And finally, you can use existing security framework as a baseline and adopt them to the AI.

As an example, there are a different security frameworks outside, right? And right now, I try to prepare a framework for the ransomware attack on AI systems that I wanted to publish it free for the companies, right? And they can go to that and they can find out where is the gap.

And this is very important to help companies that improve their security posture regarding to the ransomware attack and specifically their resilience against ransomware attack with their resilience AI system.

So the bottom line is simple, to be honest. AI systems are not just, I would say the normal software systems. They are critical assets. Why? Because they have access to the data and they integrated in your infrastructure. And attackers are already targeting them that way. So what I've told, it was a basic question that any people that have to ask themselves when they are working with the AI system and they wanted to secure them and they wanted to see that they are resilient against AI ransomware on their IT systems or on their AI systems.

Host: I loved how you sort of add step by step, right? Like first you see how are your AI systems reachable? What does your inventory look like? What external models ML libraries are you using? And doing a detailed analysis and being honest with yourself, right? Like we often say that, yeah, I know that I downloaded a model from hugging face that that's all good.

Because I have seen there are 10 million downloads of that hugging face model versus doing a thorough analysis, whether it is secure or not. instead of taking it on its face value, doing that analysis yourself, like taking those steps will give you that clarity, whether you are really safe or do you think you are safe in a way, right?

So let's say now I know, let's say what are the entry points, what does my pipeline look like and what does my like all the resources, AI system resources look like? Why do you think organizations are still not able to maybe secure them? Is it lack of resources, lack of knowledge or like, yeah, what do you think is the reason organizations are not able to act on it and maybe secure their entry points?

Behnaz Karimi: So I would say the first one, I think that there is a capability gap, right? Right now, most teams focus on traditional security controls, but they do not yet have a proper tool to monitor our AI systems or validate what's coming into their pipelines, right? A lot of these frameworks are still evolving. Right now on the market, there are different I would say there are different tools that they help you, that for example, that you monitor your model, aid, check your agent and anything of that.

But because of the increasing of that, that the many companies started to integrate AI onto their infrastructure. And as you mentioned that they wanted to be growing fast they do not consider the security that they are security controls, that they are very specific for AI systems, right? And this level, we are not talking about the traditional security controls.

And the second one, I would say there is a visibility problem. AI systems… are hard to fully understand, right? So for many people, when I'm talking with people, they tell me, okay, I cannot understand what's going on. So I cannot explain that all the mathematics thwat they are behind of that, right? But I try to give them some of the information, for example, for the agentics, for their autonomous level, how it goes and so on.

So they retrain automatically relying on many dependencies and often lack your baselines. know, the baselines, the security baselines is very important. So when something goes wrong, it's very hard to detect early. You know what I mean?

So, and the third one is governance. Governance hasn't catch up. Most organizations do not track where their models come from. As you know, we may and many times right now, right? Our property validate them. And even the way people use GenAI through prompts can introduce, I would say, real data leakage risk. And put together all of these things together, this is, I would say, this isn't a small issue. It's a border.

Systematic risk, I would say that many organizations still don't fully see it. So I'm seeing that many companies started to, for example, integrate some of the agents and some of the AI or integrate their instructor with the AI, but they didn't consider any security controls for them.

So they try to put data there and they try to use it because they want to go to the market or they wanted to improve their businesses. But I would say first security, second security, third security and fourth that would be that integration.

Host: Okay, I see what you're like. I see the point right like it goes back to how many like how what type of governance you have put in place what type of security controls you have in place and it it's some I also see a relation with security culture of the organization as well right if if as an organization you want to move fast and that's your primary focus then you would always be vulnerable to ransomware attacks because your team, maybe you or your team is just looking at how quickly I can build things, how quickly I can ship things, but you are not putting enough guardrails so that whatever you are inheriting, either through models or libraries and things like that, you have done enough validation, security validation on those.

We have been speaking about ransomware attacks. Now, when it comes to when you are in like, let's say your organization has been hit by a ransomware attack, you and you are doing incident response. What do you do? Like I know that AI systems have complex, right? There are so many entry points. Not only you have to look at your infrastructure, you also have to look at your pipelines, data and prompts, models, all of that. How do you approach?

Behnaz Karimi: So that is a really good question, right? Because right now many organization and consulting they have a lot of the suggestion regarding to the how you can be safe after when a ransomware happen. But they are not, I would say I haven't seen any kind of these things for incidents response plans for the.

AI systems that when they hit by a ransomware attack, right? And this is really good questions. And this is the one of the actually my research that I'm right now working on it. But as you know, it's very hard and it challenged some something many security teams actually, I assume.

But the most of the time, the first one isolate and eradicate, right? It is a good approach. I would say is a good starting point. But if you stop there with your AI system, it's not enough that thinking about that point is very scary. know, that think about that whole of your think about a health care company, right?

That the many doctors, many nurses, they are using the AI systems for their patients, right? And then suddenly it will be didn't stop. It's very scary. And in traditional ransomware, the process, I would say it was, as I mentioned first, that it was straightforward, right? You isolate the infected system, remove the malware, restore from backup, and you go to the normal business, right? What you are restoring is relatively simple. Files, databases, configurations, and you can verify them and move on. It's finished.

But about AI systems situation is very completely different. An AI model isn't just a file. It's built from millions of parameters shaped over time by training data. This is a very important point that always emphasizes about it this is where it gets tricky attackers do not just encrypt things they can they can also I would say quietly damage the model itself this is called poisoning so at this point your model might already be behaving incorrectly long before you even notice the attack right think about that ungoverned outcome of your model and so do you know that where the attacker came and do you know from which time point that for example your model started to produce the data that the most of the company they do not govern that they're out with.

So that means when you finally isolate and clean the system the real damage may have already happened and you may not even realize it. So what should we do instead of that? Right? For example, I told that okay the catastrophic will happen.

So first, you cannot just restore and assume everything is fine, right? You need a way to verify your models. Things like tracking where they come from and using techniques to confirm they haven't been changed. Without that, you're basically guessing, guessing.

Second, your investigation has to go deeper than usual. It's not just about, for example, checking logs and network activity, right? This is the traditional one that we are doing for the ransomware attack on AI systems or any attack. You also need to look at how the model was trained, what dependencies is used and how it has been behaving over time. You are trying to rebuild the full history of that model.

And the third one, I would say this is the hard part. If you are not fully confident the model is clean, you shouldn't restore it and you mustn't restore it. You should, I would say, retrain from scratch in a clean environment using trusted data because the model can look fine on the surface, but still behave in compromised ways that are hard to detect.

So for your question, I would say directly isolated and eradicated that we did or we are doing for the IT systems. It's not working after that for the AI systems, right? It helps you contain the problem, but with AI system recovery requires much, much deeper verification and analysis.

Sometimes most current incident response processes aren't fully prepared for yet. And this is again another gap, right? And another gap the attackers know that and attackers actually take advantage of.

Host: So, there are a couple of things that you highlighted which stood out for me at least. think incident response pre-AI world was difficult already. And now with the AI world, has become even more complicated, I feel. And I empathize with the frontline security engineers who work on incident response.

Because it's a complicated process now. More importantly, like the example that you gave where if the model itself is poisoned, that means you cannot just restore to a one week old model and you can start using it. You cannot do that. It's not just like restoring a database and you are done. You might have to retrain the whole thing, which means losing a lot of money that you have already spent. You have to spend again.

And it delays maybe your product launches and all of that, right? Like it has a lot of impact as to the business and the other thing that you highlighted which I really liked is around the output Monitoring like yeah, most organizations just look at what prompts are coming in like how do we do prompt injection security for prompt injection? But what type of data is going out of my system?

If you do not analyze, there is no way to find out if your model is poisoned or not. So looking at the output analyzing that and then this sort of determining whether your model is partially poisoned or some other entry point is poisoned so that you can only apply patch and move on versus completely retrain and start a fresh in a way. So yeah, like I can imagine the complexity that it brings to the incident response teams.

Behnaz Karimi: Very cool. Yeah, exactly. This is very a high responsibility. And I would say the high pressure on the team, to be honest. the scary point is that you don't know at which time or when your model is poisoned and your, this is very scary. And when I'm thinking about the, healthcare companies when I'm thinking about the financial institutes or this type of the company is more scary I would say.

Host: Yeah, yeah, so I'm trying to think right when I go to my doctor's office My doctor always asks are you comfortable me recording this conversation for Future use or whatever like are you comfortable in recording this conversation? I often say yeah, go ahead. That's absolutely fine now that I think about it like what if the data the model is poisoned now? My data is also getting fed in and my health care data might leak right out of it.

So yeah, I can totally see the complexity that the whole AI systems bring and how it helps the ransomware attackers anyway. So we started with or Like how you like what does your day look like you said you read a lot you understand what's going on and then you help founders and others to maybe build secure systems.

That's where you are a very active member of OWASP AI exchange, where you are sort of reshaping the security standardization for AI security. Where does OWASP AI exchange suggest that we start our risk mapping? Like, where do we do our risk analysis? How do we do it? What do you generally recommend?

Behnaz Karimi: So I would say that OWASP AI Exchange is one of the great framework for the AI security with more than 300 pages right now on the market. And we have a liaison partnership with the UAI AG. So it suggests, I try to start, keep it simple and start with the facts that AIX emphasize on it.

So start with, Now what AI you are using, build an inventory of your tools and use cases. Then you have to identify the risks that actually apply to you, not everything, just what matters. And after that, assess how likely those risks are and what impact they could have right?

And we don't want to, for example, you don't want to think about all this scenario at the time. Right now, you have to find your use case and identify the risk and assess that how likely those risks are.

And after that, plug this into a simple structure like govern, understand and manage. And most importantly, which I mentioned that this isn't a one-time task continuously. You need to check every time. You need to keep reviewing risks as your AI systems evolve, right? Any changes that you are doing on your AI systems you should again do all these steps again and this is continuously and if you want the bare minimum now what you have and understand the risks around the debt and this is the very important point and as I mentioned continuously checking your system with any changes that you are doing on your system.

Host: Yeah, I like the framework that you shared, right? Like you look at your tools and the use cases that they're trying to solve. Because often we just say that, I need a tool and maybe you don't have a use case and you just start using it. Go through your risks. What possible occurrence it might have and what impact would it have? a very simple framework to follow to determine where to focus when it comes to security. So yeah, that's an amazing framework. Thank you for sharing that.

Host: I have one last question and this question has come from Maryam is like where do you see security technology in Germany and the market for AI security in general in Europe looks like?

Behnaz Karimi: So as I'm working with the global, I would say companies and I'm working with the different founders and co-founders right now in OWASP AI Exchange, I want to answer these questions, this question as a global answer, not directed to the German market.

So what I'm seeing right now is many organizations started to integrate AI in their environment. They are using, for example, some of them they are using to, for example, generate SQL code.

And for example, some of them are using that to prepare some information regarding their data. for example, for example, bring, they are using agents, for example, bring different sources for their users in the company, right? What I'm seeing is that the market is growing and growing, but what I'm seeing here right now is that there are many gaps.

Regarding to the true knowledge and what we are, I would say what we are seeing. What I mean with the true knowledge, true knowledge, mean that, for example, you know what's going on with your AI system when something goes wrong.

You know that the best practices, if you wanted to govern your AI systems, and for example, you have a very, I would say, knowledgeable leader at the top of your project, for example, they can govern, and they can help the engineers to follow the rules that are important against any attack that they are outside.

I would say it's acceptable that, for example, or I can completely understand that many companies, they want to go to the market as soon as possible, but I do not recommend that you do not have any security controls or security consideration in your, I would say check lines when you wanted to using AR in your environment or for example, you try to push data in your AI systems or for example, do have an agent and with this agent try to communicate with each other around the globe, right?

So, they can, I would say, I just wanted to add this point, they can communicate with each other to do a ransomware agent system. So if you are thinking like that, you try to bring the safety and security at the first place. But as a summary, when I'm seeing the right now, the market the security is not considered right now at the first level, I would say. And in future, my expectation would be that people bring that on table more.

Host: Yeah, yeah. So I do hope the same thing because like we are also a security company, but we do when we work with customers or partners, we hear that business comes first and security is second. And that often is the reason of the security attacks, right? Because you have not spent enough due diligence or you do not have enough controls or governance in place to do some of the security, some of the basic security checks before you start using a model or before pushing your code to production and things like that. So yeah, I hope so too that leaders would have some controls in place before pushing AI systems to production.

Behnaz Karimi: Yeah. Right. Exactly. And the basic security controls, I would say we are not talking about the various specific security controls that would be on a deep technical level. We are talking about basic security controls, right? Because attackers, know it. And so in future, if you wanted to pay, for example, to recover your data and loss your business, business, right?

And for example, if you want to pay cryptocurrency to the attackers, this is better that right now at this place, you think as a company, you start and think about your security because it's very important because in future your reputation, your name and you lost all of them when some things happen and most important the data and you think about it.

One million of the users are outside there that they're using your system and what would be happen? So instead of that paying for attackers in future, you can, for example, invest right now from deployment phase, from the first steps, and secure your system. I would say this is the best strategy.

And one point is that we cannot guarantee that, for example, we are 100 % secure. For example, when I'm talking about the ransomware, we cannot guarantee and we cannot say that, OK, we are 100 % secure against, for example, Acura ransomware.

But at least I can say you that, okay, with these 75 security controls that you have in place, I would say maybe you have 72 % secure or resilience against, for example, Acura ransomware. For example, right now we are seeing clean ransomware, right? Clean right now is take the jump, and this is the first. But yeah, this is that talking about different people, different companies about it.

Host: So yeah, like take a moment to analyze your security posture before you start running in a way, right? Like if you want to somewhere, right? So yeah, absolutely. I hope that's the takeaway that our audience also gets out of this episode that even though you want to move fast, we are rooting for you, but at least take a moment to analyze your security posture. That brings us to the end of the podcast.

But before I Let you go. have one last question. Do you have any learning recommendation for our audience? It can be a blog, a book, a podcast, anything that you would recommend to our audience.

Behnaz Karimi: Right. So first, if they wanted to go to the ransomware topic regarding to the leveraging AI to do ransomware attack or for example, ransomware attack on AI systems.

You can read my last article about that published and Yuvi and I, we emphasize all of the points on it and I would say it's a very valuable resource because we worked on it for almost more than a year, right?

And on the other hand, I try to prepare some materials and I try to prepare some information regarding to that on my newsletter as well and my channel as well. But on the other hand, regarding to the security controls for AI systems, I highly recommend go to the OS AI exchange and read actually there or for example, AI bomb that it helps you that the basic security to understand the basic security controls.

And on the other hand, if you follow my news regarding the specific security controls for AI system that helps you, that you increase your resilience against ransomware attack that helps you that, for example, you start your AI and improve the security of your AI system against ransomware attack.

Host: Yeah, thank you so much for sharing. what we'll do is when we publish the episode, we'll add them to the show notes so that our audience can learn from these resources as well. Yeah, thank you again, Benaz for joining today and for sharing the knowledge because there were quite a few things which I was not aware of around ransomware. How does it work? What do attackers focus on? What are entry points? So yeah, thank you so much for taking the time and speaking with me today.

Behnaz Karimi: Yeah, you're welcome. Thank you too.

Host: Thank you and to our audience, thank you so much for watching. See you in the next episode. Thank you.