At CAMLIS Red 2025, we introduced BlackIce, an open-source, containerized toolkit that bundles 14 widely used AI security tools into a single, reproducible environment. In this post, we shed light on the inspiration behind BlackIce, outline its core capabilities, and share resources to help you get started.
why blackice
BlackIce AI was inspired by four practical challenges faced by red teamers: (1) each tool has a unique setup and configuration that is time-consuming, (2) tools often require separate runtime environments due to dependency conflicts, (3) managed notebooks expose a single Python interpreter per kernel, and (4) the tool landscape is large and difficult to navigate for newcomers.
Inspired by Kali Linux for traditional penetration testing, BlackIce aims to let teams focus on security testing by eliminating setup hassles and providing a ready-to-run container image.
what’s inside
BlackIce offers a version-pinned Docker image that bundles 14 selected open-source tools spanning responsible AI, security testing, and classical adversarial ML. Displayed through a unified command-line interface, these tools can be run from a shell or within a Databricks notebook that uses the compute environment created from the image. Below is a summary of the tools included in this initial release, along with their supporting organizations and GitHub star counts at the time of writing:
| tool | Organization | Stars |
|---|---|---|
| LM eval harness | eleuthar ai | 10.3K |
| Soon | Soon | 8.6K |
| Chaturhamsa | Cleverhans Lab | 6.4K |
| Garak | NVIDIA | 6.1K |
| Art | IBM | 5.6K |
| Giscard | Giscard | 4.9K |
| CyberSecEval | meta | 3.8K |
| pyrite | Microsoft | 2.9K |
| easyedit | ZJUNLP | 2.6K |
| promptmap | N/A | 1K |
| fuzzy ai | CyberArk | 800 |
| Flickering | traces of bits | 560 |
| Foul play | Dreadnode | 380 |
| judge | quotient ai | 290 |
To show how BlackIce fits into the established AI risk framework, we mapped its capabilities mater atlas and Databricks AI Security Framework (DASF). The table below shows that the toolkit covers important areas such as prompt injection, data leakage, hallucination detection and supply chain security.
| blackice capacity | mater atlas | Databricks AI Security Framework (DASF) |
|---|---|---|
| Early-injection and jailbreak testing of LLM | AML.T0051 LLM Prompt Injection; AML.T0054 LLM Jailbreak; AML.T0056 LLM Meta Prompt Extraction | 9.1 Prompt injection; 9.12 LLM Jailbreak |
| Indirect instant injection via untrusted content (for example, RAG/email) | AML.T0051 LLM Prompt Injection (Indirect) | 9.9 Input Resource Control |
| llm data leakage testing | AML.T0057 LLM data leak | 10.6 Outputting sensitive data from a model |
| Hallucination Stress-Testing and Detection | AML.T0062 LLM Search Hallucinations | 9.8 LLM Hallucinations |
| Adversarial Example Creation and Evasion Test (CV/ML) | AML.T0015 Avoid ML models; AML.T0043 Craft Adverse Data | 10.5 Black box attacks |
| Supply-chain and artifact security scanning (for example, malicious pickles) | AML.T0010 AI Supply Chain Agreement; AML.T0011.000 Unsafe AI Artifacts | 7.3 ML Supply Chain Vulnerabilities |
how it works
BlackIce organizes its integrated tools into two categories. Static tools evaluate AI applications through simple command-line interfaces and require little or no programming expertise. Dynamic tools provide similar assessment capabilities but also support advanced Python-based customization, allowing users to develop custom attack code. Within the container image, static tools are installed into separate Python virtual environments (or separate Node.js projects), each maintaining independent dependencies and directly accessible from the CLI. Alternatively, dynamic tools are installed into the global Python environment, with dependency conflicts managed through a global_requirements.txt file.
Some tools require minor additions or modifications to the image to seamlessly connect with the Databricks model serving endpoint. We’ve applied custom patches to these tools so they can interact directly with Databricks Workspaces out of the box.
For a detailed explanation of the build process, including adding new tools or updating tool versions, see Docker build README In the GitHub repo.
get started
Available on BlackIce Image Databricks’ Docker HubAnd the current version can be pulled using the following command:
To use BlackIce in Databricks Workspace, configure your compute Databricks Container Services and specify databricksruntime/blackice:17.3-LTS As docker image url in docker menu while creating cluster.
Once the cluster is created you can connect it to demo notebook To see how multiple AI security tools can be orchestrated in a single environment to test AI models and systems for vulnerabilities such as quick injection and jailbreak attacks.
check us out GitHub repo To learn more about the integrated tools, find examples of how to run them with Databricks-hosted models, and access all Docker build artifacts.
For additional details on the tool selection process and Docker image architecture, see our CAMLIS RED PAPER.
