Witness Attack: AI Red Teaming 

Witness Attack is our Simplified solution for AI vulnerability testing.
Witness Attack enables you to easily test your LLMs against common attack techniques, as well adding your own custom tests.
Out-of-the-box integrations with popular LLMs makes it simple to integrate with your self-hosted LLMs.

Available now for Apple Silicon. Download this compressed file. Open Finder to your default download location, and double-click on the “Witness+Attack-darwin-arm64.zip” file.

The uncompressed file name is “Witness Attack” or “Witness Attack.app”.

Move this file to your Applications folder. Double-click on the Application to launch Witness Attack.

Available now for x64 Windows. Download this compressed file. Open Windows Explorer to your default download location, and double-click on the “Witness+Attack-1.0.0+Setup.zip” file.

The uncompressed file name is “Witness Attack-1.0.0 Setup.exe”. Double-click this file to install.

Depending on your security settings, you may see a few alert pop-ups.

For example:

Click the “More info” link, and you may see another pop-up.

Click the “Run anyway” button, and you may see another pop-up.

Click the “Install anyway” button, and Witness Attack will launch.

The installation may automatically open Witness Attack. If it doesn’t, start Witness Attack by double-clicking the “Witness Attack” app in your Applications folder.

The first time you launch Witness Attack, a “Terms of Use” dialog will appear. Review the Terms, enter your email address, and click “Accept”.

This should not appear again on the same computer and browser combination.

To configure and run Witness Attack scans, click the Configure Scan tab and make your choices in each section.

Cards for popular models are already configured. Just add your API key. Choose any single Model to scan by clicking on it’s card. Configure the Custom API card’s API Spec to scan your own in-house models, or models that are not included in the models list yet.

Be sure to include the API parameters you want in the API Spec. They can be found on the model’s website, in the API documentation.

Parameters let you choose your maximum budget for tokens used.

Datasets are popular lists of attacks, compiled by various vendors and teams. You can choose one or more datasets to run in a scan.

You can add test datasets in this section, formatted as .csv files.

On the Configure Scan screen, when you click on any model, the pre-configured API endpoint and parameters appear in the Edit API Spec text area for each model.

The correct parameters for each model are already populated, all you need to do is add your API key after the ‘Authorization: Bearer ‘ string, replacing the placeholder text. “XXXXX” and “$GEMINI_API_KEY” are examples of placeholder text strings.

Edit any other parameters if needed. Add your API key, then click the Verify Integration button. Your edits will be saved for the duration of the scan.

Clicking on another Model, or exiting Witness Attack, will reset the API Spec to the default. Your API Key will not be saved or stored for future scans.

The circle to the left of the Model title text will turn green and display a checkmark. Verified will be displayed to the right of the Model title.

When you’re finished configuring your scan options, click the Run Scan button at the bottom of the screen:

While the scan is running, a red Stop Scan button is displayed. The bottom section of the screen shows the running status of each dataset.

When the scan has finished, you can review the Scan Results.

Clicking the Scan Report tab displays the Performance by Dataset chart.

The percentages shown at the right end of each bar indicate the percentage of attacks that succeeded.

The Datasets are displayed live, as they run. The Progress bar indicates the percent complete.

You can stop the scan anytime by clicking on the Stop Scan button.

The live view includes the Datasets as they start, Successful Attacks, Number of Tokens, and Approximate Cost.

The full list of Successful Attacks can be downloaded with the Download Successful Attacks button, or cleared with the Clear Successful Attacks button. At the bottom of the page is the pagination widget, to navigate through all pages of results.

The Logs tab provides the full log file for each Dataset run in the Scan. Individual Modules can be viewed by choosing them from the Module drop-down.

The Download Logs button downloads the individual log file for the Module chosen in the drop-down. Clicking Download Logs when “All Modules” is chosen in the drop-down, downloads a single log file containing all the logs from the scan.

All log files are formatted in JSON.