Security testing of GenAI systems is often reduced to “LLM red teaming”: probing a model in isolation to see what unsafe or offensive content it will generate. In practice this approach falls short. As security practitioners we need to assess complete LLM application use cases, focusing on how inputs and outputs propagate through application logic and enable concrete security risks such as data exfiltration, cross-site scripting, and authorization bypass.

In this talk we share practical experience from client engagements and the open-source framework we developed for this work, Spikee. Spikee is used to assess LLM applications where the model is embedded in application logic rather than exposed as a simple inference endpoint.

The talk covers:

• Approaches for testing non-conversational GenAI workflows, WebSockets, and custom APIs
• Building scoped prompt injection datasets aligned with application logic and engagement constraints
• Applying effort-based jailbreak techniques (e.g. anti-spotlighting, best-of-n, crescendo) to evaluate guardrail robustness and demonstrate practical bypasses
• Conducting meaningful testing in isolated or air-gapped environments