Product Security at Scale: Minimizing Friction & Defending AI Integrations with Sana Talwar

TLDR;

• When engaging with Product / Engineering, 3 principles go a long way. Use business language over technical jargons. Highlight Actual risk vs all risks. Automate processes to reduce burden.
• Product / Engineering teams should engage security team early in SDLC process to avoid difficult conversation before go live.
• For AI Workloads, follow few principles like red teaming of agents, prompt validation, output validation, set guardrails and most importantly have provisions for Human In The Loop.

Transcript

Host: Hi everyone, this is Purushottam and thanks for tuning into ScaleToZero podcast. Today's episode is with Sana Talwar. Sana is a Product Security Engineer at ServiceNow. She is also a speaker and an active educator in the world of cybersecurity. She currently serves on Technovation's associate board and volunteers with various women in security orgs. And beyond her technical work, she is also an active advocate for making cybersecurity and tech more accessible.

Thank you so much, Sanah, for joining me today in the podcast.

Sana: No, thank you so much for having me.

Host: I know that I did a very short intro. Anything you want to add to your journey?

Sana: Think you covered everything. think security was very, I was very lucky to find it. I knew I wanted to be in security when I was like in high school. So I think around 10th 11th grade. So it's been very grateful that I'm actually being able to be here now.

Host: It's great to hear that you knew from high school that you want to get into cyber security. That's often not the case for most security leaders. So I'm glad you had that very clear. So now that you're in security, before you get into the meat of the episode, what does a day in your life look like?

Sana: Yeah, so I think security is very varied and my day to day is also very varied. So, and I love that. So a typical day might include like reviewing security assessments, vendor assessments, or coordinating with pen test firms and engineering teams on when testing should start, what the scoping of a test should look like, and increasingly building automations for security workflows using AI.

So like tools, I think one of one, one I'm working on right now is analyzing pen test reports to auto file vulnerabilities. So it kind of removes that manual process. So it's a mix of like risk assessment, stakeholder communication, and increasingly using AI to scale and automate things.

Host: I hope we can cover some of those as part of the podcast today. So we want to focus on a few things like practical product security tips, third party reviews and AI driven security risks, which is up in your alley. So let's dive into it.

So when it comes to security, we often, let's say, read up on blog posts and we read up about best practices and we say that, yeah, we need to do that. And we try to fit that into our organization.

And however, it doesn't fit always because each organization and each environment, each setup is a snowflake in it. So what does a developer friendly security culture looks like in practice?

Sana: Yeah, I think I noticed that problem even earlier in my career and that pattern I think is still everywhere, Security comes at the end, they test a bunch of things and they give you a list of findings, right? Like here is all the places where you messed up, so to speak. I think this frustrates engineers, also frustrates security. And I think a lot of the vulnerabilities, the big problem here is that they just stay, they don't ever really get fixed. So I think there's three core principles into making a developer-friendly environment.

So number one, think would be like translating to engineering language, right? So me coming to a developer and develop like a development team and being like, you have a critical, you know, SQL injection vulnerability or a CVSS 9.1, right? Like very high scores. That's a bad approach, right? Like those like words sometimes don't mean anything.

The score doesn't mean anything. I think like a good or a better approach would be like, Hey, this buff that we found lets you like, you know, let's attackers access customer data with or without authentication. And then I think engineers really understand user impact and not just scores or high severity and things like that. I think the second principle I would say is prioritizing on actual risks and not just severity and scores.

So CVSS is very useful, but I think it doesn't always capture the business context. A good example of this is, let's say there's a high critical vulnerability but it requires you physical access to a data center versus a medium vulnerability that anybody on the internet can exploit.

So I think you always have to ask when you're giving findings to developers, what's the actual exploit scenario? How likely is it? And what's the business impact of it? And then also just in general, making developers like us more is like automating all the boring parts. You can build AI workflows to translate your plan test reports into more contextual warnings to developers. You want to auto-extract actionable items for context for engineers. Basically, think removing the friction from the fixing process can be what the real shift is.

Host: We started with a great set of principles. So I think the three things that you highlighted, right? Those resonate really well, because if you go to your engineering and say, and say that every week you go to them and say that, hey, there are 10 vulnerabilities, nine and higher. They'll be like, if everything is nine and higher, do we ignore all of them or do we work on all of them? How do we prioritize that? Right?

That's when you mentioned about context plays a role and automate as much as you can so that you have enriched context, which helps the team to understand more about the vulnerability that maybe you are passing in. And the first thing that you highlighted, like the language, right? If you just say that, hey, this vulnerability is more than nine or this has a very technical explanation, I don't care, right? Unless you tell me exactly the code that I've written has SQL injection and can get to customer data. I'd be like, OK, you just told me something technical, it's all right.

So with that, you mentioned about friction, right? These things bring friction. How do you then move beyond some of these without just handling the vulnerabilities? You mentioned about context, maybe using a language. Any other tips that you have for our audience to apply while maybe talking to their product or engineering team?

Sana: Yeah, I think this has been said time and time again. think security can't just be like a gate at the end, right? It needs to be like a handrail, so to speak, throughout. So security should be involved in planning discussions, understanding the roadmap, understanding like pressures, right? I think development teams and engineering teams have a lot of pressure to, you know, ship functional products, right? And if we make security at the end of it, like they built this entire amazing flow, right?

At the end of it, we come back and be like, actually, you need to fix most of it, right? That is not helping anybody. helping thinking about security from the beginning, right? Let's say if we know that they're building like a new API, let's talk about authorization from the beginning, like, you know, rate limiting input validation. Let's talk about all these features and all these additions we might want to think about in the security context from the beginning.

And I truly believe like, nobody wants to write like insecure code, right? like most development teams are under pressure to ship. So as long as we can make this like the secure path also like the easy path, like clear guidance, good tooling, minimal friction, I think everybody will take it. But if we make security more painful, you know, they'll work around it and that's way more dangerous.

So I don't think like developer friendly security is lowering the bar. I think it's like just making the bar like more visible, reachable, and then like integrated into existing processes.

Host: make sense. one thing is often when you work with product or engineering teams, you have to, as you highlighted it, there is possibility of friction. So you can automate, can tailor the language, you can do all of that. But that also sometimes has a cost, right?

So this is one of the questions that we got from a common friend, Enrique, was asking, like security often involves trade-offs between velocity and user experience. Then, when speaking with product teams, how do you navigate those conversations with product and engineering teams? And if you have an example where you had to push back or find some compromise, that will help our audience.

Sana: Yeah, I think again, kind of like in the business context of things, right? Explaining why a security vulnerability would affect like real life customers or like real life, like business deals really make the, make the difference on whether something is going to get fixed or not.

I think most of the times when I've received like, you know, feedback of like, Hey, this is not, this is never going to get fixed. some of the compromises I can think of was like, I, I have to come in and be like, provide solutions, right? Like if A doesn't work, provide B, C, D, right? And talk about timelines as well, right? If we don't do A, we'll do B, but it will only extend the timeline by two months or three months or three weeks. And those are the risks that you have to accept.

So coming in with like alternative solutions as well will really help with the velocity. And then there's times where you just need to escalate things. There are times where like you know that this is gonna be a problem. You have to bring in your leadership and have that conversation up top and that's what you have to do.

Host: So that means you have to like as a security owner, have to find that balance that at what point maybe you are allowing maybe the vulnerability to even go to production. And at what point you are like, no, it has to be completely blocked. Right. And use your tools as you highlighted, like use leadership or different solutions, different approaches, how you can resolve it. But you have to take that call that where do you put your foot down in a way that hey, this has to be done versus where you are flexible.

Yeah. That's great. Now the question is like we spoke about vulnerabilities, right? And nowadays we use a lot of open source libraries, third party code or services. Even in the age of AI, we use a lot of two MCPs and things like that. Now with that, sometimes even the first party code feels like third party.

And when you are doing a review, let's say somebody is using an open source library or somebody is using an MCP. When you are doing that review, what's a red flag? If you are going through, let's say, the documentation of the GitHub repo or the documentation of the MCP or a skill, what's a big red flag where you would be like, no, no, no, we cannot use it?

Sana: Yeah, I think there's a little bit of like a fundamental shift with like programming now. I think even when like a developer uses a tool like Copilot or ChatGPT to write like a function, that code is technically like third-party in a way, even if they, you know, prompted it or they, you know, merged it or whatever it is.

So I think in like third party vulnerable, like in libraries specifically, some of the red flags that are very obvious are like, are the libraries like maintained, right? Like who's maintaining them, the number of contributors that are there.

I think it's a big red flag that you should always look at. Not necessarily saying it's a security issue, but it's a big red flag when it's only a single maintainer. And that's because, you know, there's much easier chance of social engineering something. That's the number one red flag, I think. I think you also need to look at any of the PRs coming in on like a GitHub repo, right? Like what are they saying? What is being actually fixed and what actually gets merged?

I think that's always interesting. then like a quick scan of like when a before you merge something in, right, just fork the repo and have like even just like a basic security scanner against it and see what your risks are. Nine times out of 10, a lot of the times I've seen like third party libraries, if they're not like, if it's not like a foundation, you know, or like a big company maintaining it, if it's just like crowdsource, they will have a lot of security vulnerabilities, right? Most people just write it to make a, to think, you B to work.

But that's like a risk that you just have to understand. So you do a quick scanner against it. You figure out what your risks are, and then you decide whether you accept it or not. And sometimes it's accepted, and that's fine. But you have to really see what you have to do to do due diligence before merging something.

Host: Yeah, that sounds like a very good practice to apply. So when you are using any third party services or third party libraries, I think there is a project, I think OSSF scorecard or something which looks at these open source libraries and gives a particular score based on how often PRs are raised or issues are raised, how often they are fixed, maintainability as you highlighted.

It's not like you are using a repository which the last checking was three years ago, right? That means it's clearly not maintained. So yeah, you have a very good suggestion of following those practices. Do you follow something similar when you review something which is generated internally or you primarily use that for external third-party code or third-party reviews?

Sana: Yeah, think internally we should, I think any code that's even AI-generated internally should go through a conducive security reprocess or a comprehensive security reprocess, no matter how it's generated. You're building a product, especially if it's like bigger, large scale companies, right? You're shipping it out to like hundreds of thousands of people sometimes.

So even if it's AI generated, doesn't matter. It needs to seem like it came from like a, you know, a developer. I think some of the red flags that you should look out for in that case is like if a developer can't explain like what is going on in their code, right? If we ask them like why we implemented something a certain way and the quick answer is like the AI suggested it and it worked. I think that's a red flag.

I think developers really need to understand what they're building and like what they're building. There's also like a lot of with AI generated code unexplained like complexity. I think AI generated code like loves to over engineer problems.

So abstract factories will have, for simple functions, have, it can cause a lot of technical debt and security holes. And also AI generated code does not have a lot of context on data handling, right? You know, as a developer, what data is sensitive and how it generally should be stored. But the AI doesn't really know your policies or it might log sensitive data or cache credentials or things like that.

And it doesn't know what sensitive in your context, no matter. don't think it's sophisticated enough to like get there yet. think it probably will be soon, but not there yet. And there's also like a lot of dependencies that come with AI generated code. I've seen, I've built a lot of things just for fun on cloud code, and it will like download a library to use one of the library.

Right. Is it really worth it to have all that, you know, unknown expectations? Like, I think that expands your attack surface like dramatically with just importing library after library. So I think that's really like, I think it's interesting. I think we need to think about it as any AI generated code is generated from like right now, like a junior developer that you don't trust right now. So you want to review it thoroughly. You want to review it comprehensively. And you have to also account for the developer as well, right? Can they intellectually, do they intellectually own it? And do they understand it well enough to fix it or debug it? You know, I threw you up.

Host: You touched on a very important point which is ownership, right? Just because you used AI, Cloud Board or ChatGPT Gemini or whichever tool, To generate some code to maybe fix a bug or build a feature. The ownership still lies with you. You cannot just say that, hey, AI generated code and I have no idea what it is. You cannot say that, right? You own that code now and you have to answer to those questions, which we do even for non-AI generated code, right? It goes to the review. Somebody might say that why we have written it in a particular way versus some other way so you have to answer to those questions and you have to get those reviewed before it goes to production.

Sana: Yeah, no, exactly. I want to be clear, I'm not anti-AI tools. I use them daily. But I think like developers and like security engineers need to like adapt the review process, right? Like everybody needs to understand why they built something and how to maintain and secure it as well.

Host: Yeah, sorry if it came out as that I'm anti AI because I also use AI quite a bit to generate code and things like that. But at end of the day, I sort of own that code that is being written by the agent that I'm using. And I make sure that I understand and then I ask for thorough review before it goes to production. yeah, like spot on what you said, use AI as much as you can to generate code or generate specs and all of that. But at the end of the day, you have to understand that deeply so that you can answer questions or tailor it if there are some review comments and things like that.

So on the same lines, one of the questions that we have got from Vineet is that when it comes to third-party libraries or things like that, we often obsess on S-Box. But most of the times, if it is a vendor-owned product, it's a black box to the security team or the product team or engineering team. When there is a zero day, then S-BOM just tells us that what component is used, but it doesn't tell us whether the vendor has fixed that, mitigated that risk or if we are exposed or not.

In this AI driven world, do you think AI can finally help us with that point in time assessment? How do you see it?

Sana: I think a lot of vendors are already doing SBOM assessment. you know, generating, providing like a vex to customers, right? Like what's going on with all the vulnerabilities figuring out the criticals and what is actually affected. So, but even like, I think the problem with the SBOM is that even if you have an SBOM, right? And it says like, this library is vulnerable on this version and you're using it, doesn't necessarily mean that you're vulnerable.

You might not be using that function in the library, right? Or it's really hard to exactly know from an SBOM what is being vulnerable or like what is putting you at risk and what's not. So think doing like SBOM assessments on your like, you know, whatever release cycle that you have, whether it's automated or doing it manually, I think it's really important to get that context.

And I think AI can do a lot of help with that, right? Like let's say your SBOM for a month ago said, version 1.0 and now you're already upgraded to 2.0, right? Now you're not vulnerable anymore, but the SBOM is outdated. So I think AI can be really useful in reviewing your repositories and updating the SBOM. But I do think assessments of SBOMs, right? To see if any of them are actually exploitable is really important.

Host: And I think, again, the key, the keyword that you use is exploitable, right? You might use a lot of, sorry, exploitable. And another thing that you highlighted is whether you're actually using them or not. It's possible that in the library that you're using, there are 200 functions, but maybe out of those only, you're using 10, which are in your flow, code flow, and they are, they are not vulnerable, right? So yeah, like doing that thorough review makes a lot of sense.

One follow-up question to that is, let's say you highlighted about the upgrade, right? Let's say you initially installed 1.0. Now there is a version of 1.1. As a security person, how do you ensure that 1.1 is not an malicious update without going through maybe all the changes or all the change logs and the code and everything. How do you ensure that the update that or the upgrade that we are going to do is safe?

Sana: Yeah. Yeah. So I think, I think that's difficult problem to solve because you can't review like every line of the new, you know, the new version. That's not viable. Doesn't scale. And if you could like, you probably also like, like anybody would also like lack the expertise to audit everything.

Like I can't personally audit like a crypto library. I don't know. You know, I'm not in depth with that. I think managing those malicious updates are like, you know, that that risk responsibility is you have to think about it in like layers, like defense in depth, like which is a core cybersecurity principle, right?

So in this case, you could think about things like having, you know, an S-BOM, making sure you know what's going on with this new upgrades. Dependency pinning, right? So never auto updating in production always goes through staging first, making sure things are looking good before you push it out to prod.

There's a lot of like good automated scanners that use it that, have a lot of good tooling to flag any known vulnerabilities. And then also just evaluating, I think I mentioned this before, project health. What's the active maintenance, contributors, and corporate backing? What's the deal with the new update?

Another thing I think you can look at is the runtime monitoring of things in terms of detection and containment, detecting any anomalous behavior, always following least principle, like limit what dependencies can access, whether it's like sandboxing or just like, you know, access control measures. And then the easiest one, I think is like a signature verification, right? Make sure if packages, the signature and the checksums match, right? You make sure you're downloading what you think you're downloading.

Host: I think the, again, in this, key thing that you highlighted is it's not humanly possible, right? For an organization to stay on top of all the third party libraries you're using. And again, all the upgrades that are coming in every single line. The example that you gave of, let's say, crypto library. Maybe you are not that familiar with crypto.

What about it's written in a language which you don't understand? let's say I don't understand Rust and I'm using a Rust package and there is a change. There is no way I can understand what exactly the code changes are, whether it's vulnerable or not. That's where, you highlighted, using tools to do some of the scans and automating this as much as possible would give you that visibility, that maybe if we upgrade, what are the risks are we agreeing to take? And maybe then you decide whether you want to move forward or not.

Now, one last question on this is, let's say you are using a library and you see that there is a potential vulnerability or it's not maybe maintained really well. One of those red flags that you highlighted, but maybe one of your product managers or engineering manager is like, is solving my problem and I need it. I don't care about what you're saying. So how do you communicate? Like, how do you find balance in that scenario?

Sana: Yeah, I think I'm going to go back to what I previously said about communication. So as I mentioned, like security people speak vulnerabilities, we speak severities, and PMs speak like business outcomes and like timelines. And those are completely like two different conversation. So I could come in and be like, hey, this new library that you're using has like very high critical CVEs attached to them. But that doesn't really mean anything to PM. Like PM wants to like PMs want to know like what risks are they getting and what are they going to accept?

So like a three-part framework that I have for this is, you know, if something is going to be like a compliance broker, right? Known vulnerabilities in libraries or in anything will fail like SOC 2, for example, right? And if your customers rely on you providing like a clean SOC 2, that's a very easy way to communicate risk.

Another thing is like customer trust, right? Like if we know that these are known vulnerabilities that, and we didn't fix them and we still use them, that shows negligence and very down the line PR disaster lawsuits that could create like real, actual, like very drastic impact to the product.

And then there's also like technical debt, right? Is it easier to fix it now or like two, three months from now when more code has been integrated and it's like much difficult to migrate and refactor the code to change it. So that would then that could extend timelines on your product development, right?

I think those are the three top three things that I think, if you can manage your problem within one of those frameworks or one of those principles would really communicate the impact to your PM.

Host: Awesome, very clear framework or clear set of guidelines that you follow when interacting with a product or engineering team. For sure, that goes a long way, right?

One of the questions that Vinit is asking on this topic is, how do we perform continuous monitoring for code we don't even own without breaking the speed of modern devs. I think this goes back to what you said earlier, that if you're using, let's say, external libraries, you need to have tooling, you need to automate it.

But how do you do it in a continuous manner and not impact your maybe deployments or your dev speed? What would you recommend?

Sana: Yeah, I think that's the problem that I everybody faces, It takes a lot of time and energy and resources and continuous monitoring versus you do it at a point in time. I don't know if any tools speak out to me to do this, but you have to make the risk trade-off of what's more important, right? If you do a point in time test every month, I think generally speaking, for most products, that's very sufficient.

And unless there's no need to do continuous monitoring at that point. Or if you want to do it per release, right? Before every release, are you making sure that you're assessing, you know, whatever, whatever you're trying to monitor, right? I think it really depends on your business needs and what your customer wants. So continuous monitoring is really good if you have like a large-scale setup and you can afford it in terms of resources and, you know, the people that would actually be following up with that monitoring, like what does that actually translate into?

But if you have like a small scale operation, right, then doing it by every release or every month or whatever the cadence seems right to you is probably just fine as well.

Host: We spoke about vulnerability, third party and all that. And we touched on AI a little bit in between. Let's get into AI. So when it comes to AI tools, security teams can use it for multiple purposes. Either to use it to improve the product security itself or to improve the product itself, right? So how can, let's say, if an organization has an early security team, how can they use AI to improve the product security without maybe introducing new set of risks?

Sana: Yeah, I think starting small, right? AI is still pretty, if you're completely new to it, starting with small, high-constrained use cases would be great. Good starting points would be a document analysis, right? If you get pen test reports, sows, questionnaires, vendor assessments, right?

I think those are really easy for AI to digest and give you back and give you good feedback as well. They're really good at taking structured info or giving you unstructured information from unstructured tests. And that will eliminate a lot of manual process and create productivity gains. Another obvious thing that I see is pattern detection.

AI can spot anomalies that humans might miss in high volume environments like logs or traffic captures, things like that. I think a couple of other things to improve product capability would be making would using AI to help make security more accessible to developers.

So for example, like I was mentioning for like an AI would explain vulnerability reports in plain English, right? Nobody really needs to know what's you know, like the continent, the technical name for something, right? But like, if you translate into plain English, and like how to fix it in plain English, right? I think that would be like a great use case for developers.

I think another use case could be suggesting secure code patterns during development. Like the security focus like copilot even. So when a developer saves a file that has a password function, And AI can suggest, consider using bcrypt instead of shot256 in your hashing, right? I think those suggestions could catch a lot of the low-hanging fruit vulnerabilities very early on and very easily be able to fix it very easily.

And then threat modeling, think I haven't ventured into this, but I think threat modeling is, know, like AI would be like really useful in threat modeling, especially as like the first step, right? So you give it all your architecture documents, you give it all your documentation, and it can give you like a really good like first page for you to start on, you know, making those architecture decisions, making those changes in your, in your structure from the beginning.

Host: You shared some very good-examples of how we can use AI for security. I think the first one that you touched on like Pentest reports or vendor security assessment reports, I think I recently read like treasured data, they used agents to analyze these reports and they could get 94 % efficiency in their third party risk management. yeah, it makes a lot of sense to utilize AI to do some of these report analysis and cut down as much of repetitive tasks or trivial tasks. And then once you get the skimmed down version, then maybe you work on that, right? So that reduces the burden on the security teams more, reduces more burden from the security teams.

Sana: Correct, yeah. And like, we're not replacing security engineers by any way. I'm a security engineer, so I don't want to be replaced. We're just like multiplying, like I said, like the handling, the repetitive tasks, the documentation type of work, so we can really focus on like the novel threats.

Host: I think that's a touchy topic that whether AI is replacing the results or not. Maybe we'll get into that. But yeah, I understand. It helps with lot of productivity gain, both from coding perspective or even from day-to-day work perspective. We are noticing a lot, especially in the security industry, AI is being leveraged in many ways to improve productivity. yeah, totally agree with your point.

One of the questions that we have received from Akanksha on the same topic is, what's your take on AI performing these critical job functions? Let's say reviewing the security posture of the product and application.

Sana: There's a level of risk management there, right? So you can use AI's great efficiency and productivity, all the good stuff, but you have to manage your risk. something as simple as like, I know I shouldn't have to say it, but like never sending sensitive data to public AI services, right? You can't copy and paste your pentest report into like your free chat, your public chat GPT, right? Or your security. You don't want to send your security architecture to like a public API.

So for those, you want to manage your risk, right? You're making sure that you want to use private deployments, enterprise deployments, or anonymize any data. And I think the big thing here is, especially on critical job functions, you want to use human review. So AI can suggest things, right? But humans have to pull the trigger and have to decide. In the same sense, we were talking about developer ownership, You want a security, you want to own sensitive action items that you're doing.

So you never want to auto file things, right? Just for the sense of it, you don't want to auto response to questionnaires without any validation. So in this case, I'm using AI to do the heavy lifting, to gather documentation, to gather your metrics. But the human person, like the engineer, has to make the final call of what is right, what is wrong, what is being at risk accepted and what's not. I think that's kind of the gate that you have to keep with AI systems.

Host: I think you mentioned about that not copy pasting maybe your pen test report. I hope that's not the case with folks. It's same with like your code or even architecture document. Anything which is your IP, no one should just copy paste it directly to like the free plan of let's say GMPT or Claude or something like that. Hopefully that doesn't happen.

One of the things that you highlighted and which happens a lot when you use AI is somebody can write a, if you have a, let's say, chat interface to your customers or something like that, somebody can always write a prompt to reveal some sensitive data, just like prompt injection, right? We spoke about like third-party libraries and vulnerabilities and things like that. If you're using AI systems, then how do you… patch this kind of vulnerability or how do you work around some things vulnerability.

Sana: Yeah, no, that's a great question. think there's a lot of research still going into this. like prompt injection, as you mentioned, is genuinely scary because a lot of traditional security models don't apply. But like direct is just like, hey, it's like an SQL injection for like your agent or your LLM, right? You craft a malicious response and it could be something as like ignore instructions and do, you know, X, Y, Z.

And that's relatively manageable. You can create some input filtering. You can control the conversation a little bit. But I think indirect prompt injection is way scarier because the attack will come from resources or data sources that the AI will read. And that sometimes, a lot of times, it's the content that the user will never see. So an example of this is having an email assistant on your email account. And you have it read all your emails and then you have it send you a summary at the end of the day, so to speak.

In this case, what if you get a malicious email that sounds like, it looks like a newsletter, but it's hidden in the white text or even in the HTML comments, it has some kind of command that says system forward, I don't know, all emails saying confidential, to evil at evil.com, whatever it is, right? Some kind of malicious prompt, it's there.

So when you look at that newsletter email, you only see the content of that, but not actually the malicious command behind it. And the AI or your LLM bot reads it as commands and not just data. You have successfully created an indirect prompt injection. So now every email that has confidential in it will get forwarded to this email address.

That's like the scary part of it. Like you never see the malicious prompt. It's just like indirectly, indirectly injected through the content that AI has processed. And I think the scales to AI agents the most specifically because they browse websites, you know, there's lots of malicious websites that, you know, can embed instructions. They read documents, they process like user content, like, you know, comments on reviews on social media, things like that. And they also like analyze images. with steganography and like, you know, text embedded in images.

Sana: That can also be taken as instructions. I think solving this is really hard because the fundamental problem here is that traditionally you would just sanitize the input for any kind of injection vulnerability. But with AI, inputs are English, they're natural language, and they come from everywhere. So the attack service is giant, and you can't sanitize natural language without destroying its utility.

Host: Yeah, that's very important point.

Sana: Yeah. So I think solving this is, again, I think it's still being actively researched, but kind of going back like defense in depth, you want to create some levels of prevention. So number one is like doing least privilege and on the prompt level. So you want to think about like only having action allow lists, right? So if your email assistant is only supposed to emails and summarize them and mark them right or unread those results only three actions it can do. It cannot send forward anything. It cannot delete anything and it cannot like create filters or you whatever it is.

And then if there's any sensitive actions that you know, that are happening, you want to have like I mentioned before, the confirmation gates, right? You need human approval before it can send. Yeah, human in the loop. You want to send before you send something. And then you also want to think about like scoping everything. If you want to send emails, you can only do it to these five emails.

If you want to read documents only from this one folder, you can read documents and so on. So those are the few things that I can think of that would help like patch this. But I don't think there's like a perfect solution for prompt injections or indirect prompt injection just yet.

Host: It's an exciting world we live in, Like you drew parallels between SQL injection and prompt injection. But now with, let's say, AI agents using MCPs or using skills, the example that you gave of email, right? Let's say you are using a MCP tool to read your emails. Now it reads a command as part of it, and it becomes an indirect prompt injection, right? You need to solve for that as well.

You're right, like prompt injection, even in the worst problem of LLM, prompt injection is one of the most cited risks. And with tools going to browsers, like the example again that you gave, right? Like if you allow your, let's say, cloud or your codecs to go to browsers to get some additional details, it might get some malicious prompts from there and just execute it, right? Yeah.

Now, the… you highlighted about least privilege, setting guardrails and boundaries and things like that. One of the questions that Enric asks in the same lines is, with AI integrated across the stack, with agent-dikai and all of that, it completely changes how we look at security. What new risks should we be paying attention to? What kind of new risks should we be paying attention to?

Sana: I think the biggest shift to think about here is like we can't fix all AI vulnerabilities, like how we patch code, right? Prompt injection is a really good example of it's like how it is like so inherent to how LLMs work. So security, think becomes more of a resilience rather than like perfection, right? We always want to mitigate risks rather than just, it's really hard to eliminate that risk altogether. There's not one simple format for you to not have malicious prompts and bypass your system, right? So you want to think about resilience and how you mitigate that risk.

So we need practices like before you deploy LLM or an agent, right? Red teaming it, treating every external input as malicious. You want to validate outputs as regularly as you validate inputs. I think that's very important. We all talk about sanitizing the input, but in this case, the world of AI, output generation also needs to be very closely monitored and sanitized. And then, again, humans need to stay in the loop.

We cannot have the… LLMs and AIs do all the work for us. For sensitive actions and identifying those sensitive actions, there needs to be a human in the loop. Somebody needs to review it just for safety and security. Because AI can be manipulated in ways like we're discovering a new way every single day, I feel like. There's a new attack vector, a threat every single day.

So there's no way for us to catch them all. And the only way we can do it is building good security principles from the beginning and hope hope that it will, you know, at the very least, maintain that risk.

Host: I think two key things that I could gather from your responses, you should not think that we can mitigate all the risks that comes with LLM, like using LLM for your applications or using HNTK using agents.

The second part is the basic principles that you need to follow. You cannot skip that even though you know that there are risks, you know you have to mitigate most of that, but you cannot forget about basics. You need to do some of those basic principles. like some of the examples that you gave like red teaming your agents or doing the output validation, which is often, we don't think about that in pre-AI world, right? We just think about input validation and most importantly, keeping humans in the loop. If there is an elevated action that maybe an agent is taking it should go through maybe somebody's review, manually reviewing and approving it before agents just starts taking those actions.

So yeah, those are some amazing suggestions, amazing recommendations. I hope our audience would use those when they are building and deploying AI applications.

Before I let you go, one last question that I want to ask is, do you have any learning recommendation for our audience? It can be a blog or a book or a podcast or maybe a YouTube channel or discord, anything that you would recommend.

Sana: Yeah, I think for me, it's still relevant. The OG security gurus are still relevant. So like Krebs on security and Schneier on security. I think they've been around for so long that they're very reliable and I really enjoy all the content that they put out. I think there was a recent article that Schneier on security posted that autonomous drones are being injected by prompts and road signs.

Which is so crazy. But I think the traditional, known, Krebs on Security and Schneier on security are like my top two. I actually have like an Apple shortcut that sends me a summary of what they have posted in the last 24 hours. And I get like a text message of it every single day. So, yeah, I follow those two the most, I think.

Host: Wow, so you have automated even your learning process. So yeah, thank you for sharing that. When we publish the episode, we'll add them to the show notes so that our audience can go and they can also learn from the same resources.

Thank you so much, Anna. This was like an amazing discussion on all the way from like working using securely the third party libraries to third party applications to AI and challenges with AI and how do we stay on top of it from a security perspective. So yeah, thank you so much for coming to the podcast and sharing your knowledge.

Sana: No, thank you for having me. This was a really fun discussion.

Host: That's great. And to our audience, thank you so much for watching. See you in the next episode.