Embrace The Red

This year I spent a lot of time reviewing, exploiting and working with vendors to fix vulnerabilities in agentic AI systems.

This is a security advisory for a data leakage and exfiltration vulnerability in a popular, but now deprecated and unmaintained, Slack MCP Server from Anthropic.

When the Model Context Protocol (MCP) came out it reminded me of the Common Object Model (COM) from Microsoft.

Today we are going to discuss how real-world tactics, techniques, and procedures (TTPs) apply to computer-use systems, specifically, we’ll look at ClickFix attacks.

Recently OpenAI added an additional memory feature called “chat history”, which allows ChatGPT to reference past conversations.

The Model Context Protocol (MCP) is a protocol definition for how LLM apps/agents can leverage external tools.

GitHub Copilot has the capability to be augmented with custom instructions coming from the current repo, via the .github/copilot-instructions.md file.

You are likely aware of ASCII Smuggling via Unicode Tags.

ChatGPT Operator is a research preview agent from OpenAI that lets ChatGPT use a web browser.

Imagine your AI rewriting your personal history… A while ago Google added memories to Gemini.

At Black Hat Europe I did a fun presentation titled SpAIware and More: Advanced Prompt Injection Exploits.

I regularly look at how the system prompts of chatbots change over time.

Happy to share that I authored the paper “Trust No AI: Prompt Injection Along The CIA Security Triad”.

Grok is the chatbot of xAI.

Last week Leon Derczynski described how LLMs can output ANSI escape codes.

About two weeks ago, DeepSeek released a new AI reasoning model, DeepSeek-R1-Lite.

A few days ago, Anthropic released Claude Computer Use, which is a model + code that allows Claude to control a computer.

This post explains an attack chain for the ChatGPT macOS application.

This post describes vulnerability in Microsoft 365 Copilot that allowed the theft of a user’s emails and other personal information.

Recently, I found what appeared to be a regression or bypass that again allowed data exfiltration via image rendering during prompt injection.

Microsoft’s Copilot Studio is a powerful, easy-to-use, low-code platform that enables employees in an organization to create chatbots.

Google Colab AI, now just called Gemini in Colab, was vulnerable to data leakage via image rendering.

Recently, OpenAI announced gpt-4o-mini and there are some interesting updates, including safety improvements regarding “Instruction Hierarchy”: OpenAI puts this in the light of “safety”, the word security is not mentioned in the announcement.

Imagine you visit a website with ChatGPT, and suddenly, it stops working entirely!

This post highlights how the GitHub Copilot Chat VS Code Extension was vulnerable to data exfiltration via prompt injection when analyzing untrusted source code.

In the previous post we demonstrated how instructions embedded in untrusted data can invoke ChatGPT’s memory tool.

OpenAI recently introduced a memory feature in ChatGPT, enabling it to recall information across sessions, creating a more personalized user experience.

This post is part of a series about machine learning and artificial intelligence.

Recently Google published a blog about detecting browser data theft using Windows Event Logs.

Google’s NotebookLM is an experimental project that was released last year.

This week was HackSpaceCon 2024.

What I like about the rapid advancements and excitement about AI over the last few years is that we see a resurgence of the testing discipline!

About a year ago we talked about how developers can’t intrinsically trust LLM responses and common threats that AI Chatbots face and how attackers can exploit them, including ways to exfiltrate data.

I added a couple of features and improvements to ASCII Smuggler, including: Optional rendering of the BEGIN and END Unicode Tags when crafting hidden text Added a feature to URL decode the input before checking for hidden text Output Modes for Decoding: S…

Building reliable prompt injection payloads is challenging at times.

Last November, while testing Google Bard (now called Gemini) for vulnerabilities, I had a couple of interesting observations when it comes to automatic tool invocation.

Your Code Interpreter sandbox, also known as Advanced Data Analysis sessions, are shared between private and public GPTs.

A couple of weeks ago hidden prompt injections were discovered and we covered it at the time.

A few weeks ago while waiting at the airport lounge I was wondering how other Chatbots, besides ChatGPT, handle hidden Unicode Tags code points.

Last November Google had an interesting update to Google Bard.

A few weeks ago Amazon released the Preview of Amazon Q for Business, and after looking at it I found a data exfiltration angle via rendering markdown/hyperlinks and reported it to Amazon.

A few days ago Riley Goodside posted about an interesting discovery on how an LLM prompt injection can happen via invisible instructions in pasted text.

Five years ago I gave a Lightning Talk at the 35th Chaos Communication Congress called “Pass the Cookie and Pivot to the Clouds”.

OpenAI seems to have implemented some mitigation steps for a well-known data exfiltration vulnerability in ChatGPT.

When OpenAI released GPTs last month I had plans for an interesting GPT.

Earlier this month I had the opportunity to present at Ekoparty 2023 about Prompt Injections in the Wild, and the video of the talk was just posted to YouTube.

Recently Google Bard got some powerful updates, including Extensions.

Large Language Model (LLM) applications and chatbots are quite commonly vulnerable to data exfiltration.

Large Language Model (LLM) applications and chatbots are quite commonly vulnerable to data exfiltration.

During an Indirect Prompt Injection Attack an adversary can exfiltrate chat data from a user by instructing ChatGPT to render images and append information to the URL (Image Markdown Injection), or by tricking a user to click a hyperlink.