Adversarial AI Digest — 20 March, 2025

A digest of AI security research, insights, reports, upcoming events, and tools & resources. Follow AI Security updates on Twitter and LinkedIn group for more.

🔍 Insights

🟠 Today’s AI-Powered SOC Companies Are Tomorrow’s Security Service Providers — Ross Haleliuk explores how AI-driven SOC platforms may disrupt the MSSP/MDR industry, shifting businesses towards AI-native security operations. https://ventureinsecurity.net/p/todays-ai-powered-soc-companies-are-524

🟢 How to Judge Security Products That Claim to Use AI — Harry Wetherald highlights how AI is often a marketing buzzword in security products and provides key questions to evaluate real AI-driven solutions. https://harrywetherald.substack.com/p/how-to-judge-security-products-that

🔵 Securing AI/LLMs in 2025: A Practical Guide to Securing & Deploying AI — Francis Odum details the risks, costs, and security challenges of AI deployment and how cybersecurity can drive secure enterprise AI adoption. https://substack.com/home/post/p-158740618

🟠 Autonomous Ethical Hacking with Crew AI Agents — Toni Ramchandani explores how AI agents can autonomously identify vulnerabilities and perform penetration testing. https://medium.com/data-and-beyond/autonomous-ethical-hacking-with-crew-ai-agents-next-gen-cybersecurity-penetration-testing-f3ed0b53ff5e

🟠 How We Beat Alert Fatigue with AI — Asana’s AI Studio automates security workflows, categorizes threats, and assists with bug bounty triage, reducing manual alert fatigue. https://asana.com/resources/how-we-beat-alert-fatigue-ai

🟠 New Vulnerability in GitHub Copilot and Cursor — Ziv Karliner from Pillar Security explores how hackers exploit AI-driven coding assistants by manipulating rule files to introduce silent backdoors into generated code. https://www.pillar.security/blog/new-vulnerability-in-github-copilot-and-cursor-how-hackers-can-weaponize-code-agents

📄 Reports

🔴 Securing the UK’s AI Research Ecosystem — A report by the The Alan Turing Institute Institute and Center for Emerging Technology and Security (CETaS) analyzes vulnerabilities in the UK’s AI research sector, emphasizing risks from state-sponsored actors.

🟡 Balancing Security and Usability of Large Language Models: An LLM Benchmarking Framework — NetSPIreleases a benchmarking framework assessing LLM security vs. usability trade-offs. https://github.com/NetSPI/Open-LLM-Security-Benchmark/blob/main/open-LLM-Security-Benchmark.pdf

⚫ AI and Software Cybersecurity Market Analysis (DSIT 2024) — A report analyzing the UK market’s AI security providers, investment trends, and technical challenges. https://www.gov.uk/government/publications/ai-and-software-cyber-security-market-analysis

🟢 US Open-Source AI Governance — Claudia Wilson & Emmie Hine highlight backdoor threats, supply chain risks, and model manipulation vulnerabilities in open-source AI. -Center for AI Policy — https://www.centeraipolicy.org/work/us-open-source-ai-governance

🟠 ISO 42001 Checklist Overview by Rhymetec — A structured four-phase checklist for achieving ISO 42001 certification for AI security governance. https://www.linkedin.com/feed/update/urn:li:activity:7301267213238382593

🟠 OWASP AI Threat Research: LLM Exploit Generation — A study evaluating GPT-4o, Claude, and DeepSeek R-1 for their ability to automate security testing using OWASP Juice Shop. https://genai.owasp.org/resource/owasp-llm-exploit-generation-v1-0-pdf/

🔴 2025 Cato CTRL Threat Report: Rise of the Zero-Knowledge Threat Actor — A Cato Networks threat intelligence researcher, Vitaly Simonovich 🇮🇱 with no prior malware coding experience, successfully jailbreaked multiple LLMs, including DeepSeek-R1, DeepSeek-V3, Microsoft Copilot, and OpenAI’s ChatGPT-4o, to create a fully functional Google Chrome infostealer for Chrome 133. https://www.catonetworks.com/resources/2025-cato-ctrl-threat-report-rise-of-zero-knowledge-threat-actor/

📅 Upcoming Events

🔹 BlackHat Asia 2025 AI Summit (April 2, 2025) — A dedicated summit at Black Hat Asia 2025 focusing on AI security challenges, adversarial machine learning, and risk mitigation strategies in real-world deployments. https://www.blackhat.com/asia-25/ai-summit.html

🔹 AI Agent Security Summit (March 31, 2025) — Organized by Zenity, this event in NYC will bring together AI researchers, CISOs, and security professionals to discuss security risks related to AI Agents. Sessions will cover governance, compliance, and threat modeling. https://www.zenity.io/resources/events/ai-agent-security-summit-2025/

🔹 AI Security & Privacy Conference (26 March 2025) — This second edition of the Al Security & Privacy Conference aims to create a space for understanding and discussing these risks and challenges, contributing to a greater awareness of this unexplored new dimension. https://www.aisecurityconf.com/

📂 Tools & Resources

🔹 Awesome Safety-Critical AI — A curated list of AI security references for safety-critical systems. https://github.com/JGalego/awesome-safety-critical-ai

🔹 PyRIT — Python Risk Identification Tool, an open-source framework for identifying risks in generative AI systems. https://github.com/Azure/PyRIT

🔹 AgentFence — An open-source platform for AI agent security testing, detecting prompt injection, secret leakage, and system exposure. https://github.com/agentfence/agentfence

🔹 OASIS — Ollama Automated Security Intelligence Scanner. https://github.com/psyray/oasis

🔹 AIAPWN — Automatic Prompt Injection Testing. https://github.com/karimhabush/aiapwn

📚 Research

🔹 Dissecting Adversarial Robustness of Multimodal LM Agents (CMU, ICLR 2025) https://arxiv.org/pdf/2406.12814

🔹 Risk Taxonomy & Assessment Benchmarks for LLM Systems https://arxiv.org/html/2401.05778v1

🔹 Zero-Trust AI Model Security Using Moving Target Defense — A security model integrating Content Disarm & Reconstruction (CDR) and Moving Target Defense (MTD). — Daniel Gilkarov, Ran Dubin https://lnkd.in/grECtjFF

🔹 NYU CTF Bench: Evaluating LLMs in Offensive Security — NYU’s open-source CTF dataset tests LLMs like GPT-4, Claude 3, Mixtral, and LLaMA on 200 cybersecurity challenges. https://arxiv.org/pdf/2406.05590

🔹 Emergent Misalignment in Fine-Tuned LLMs — Fine-tuning GPT-4o for insecure code led to unexpected misalignment in 20% of cases. https://martins1612.github.io/emergent_misalignment_betley.pdf

🔹 Multi-Agent Risks from Advanced AI — Researchers from the Cooperative AI Foundation explore security threats posed by multi-agent AI systems and mitigation strategies. https://arxiv.org/pdf/2502.14143

🎥 Videos

1️⃣ The AI-Augmented SOC: How far are you willing to go? (Francis Odum) https://youtu.be/84Op47flrjw

2️⃣ Hacking LLMs Demo and Tutorial (Explore AI Security Vulnerabilities) (David Bombal) https://youtu.be/tiwx7WPW8Jc

3️⃣ How to Attack and Defend LLMs: AI Security Explained (Yaniv Hoffman) https://youtu.be/6bYGhY9HB8k

4️⃣ Living off Microsoft Copilot (Tamir Ishay Sharbat, Gal Malka, Lana Salameh) https://youtu.be/FH6P288i2PE

5️⃣ Remote Code Execution on Cursor AI and Notion (Navdeep Singh) https://youtu.be/16ZwrJ-u7gM

💬 Read something interesting? Share your thoughts in the comments.