I tried running two AI pentesters on one local model

2 October 2026 · Shannon, BugTraceAI, Ollama, and a very patient MacBook

Thirteen agent jobs were queued behind one local model. Ten minutes later, none had finished, so I stopped the run. I had wanted to see how far an AI pentest could get without paying per token or sending a codebase to a hosted model.

I had two questions. Could I make the whole setup run on one laptop? And if I gave two different tools the same model and target, would they find the same bugs? I got an answer to the first one. The second one will have to wait.

This is the story of that run, including the bits that were more awkward than I expected and the point where I stopped. It is not a benchmark, and it is definitely not evidence that a local model can replace a pentester.

The tools I used

Shannon v3.3.0 for the white-box side: it reads the source and drives the live application.
BugTraceAI-CLI 3.7.30-beta for the black-box side: it approaches the application from the outside.
Ollama 0.33.2 serving qwen3.6:35b-a3b as the local model for both tools.
OWASP Juice Shop v20.2.0 as the deliberately vulnerable target.
Docker 28.1.1 for the target and Shannon's worker containers.

Both tools talked to the same Qwen model and were pointed at the same disposable Juice Shop container. The setup looked like this:

Everything ran on one laptop. That did not make it a sandbox. Scroll horizontally to explore the diagram.

Using the same model was deliberate. I wanted the orchestration to be the variable, not the model. It also meant both tools would share the same model-specific blind spots, which is one reason this could only ever be an experiment rather than a fair comparison of their full potential.

All of this ran on a MacBook Pro with an M1 Max and 64 GB of unified memory. The model was about 22 GB on disk. The versions above matter because every project involved is moving quickly.

The plumbing worked

I started with OWASP Juice Shop because it is intentionally vulnerable and meant to be broken. I bound it to loopback so I did not accidentally advertise a vulnerable application to the rest of the network:

docker run -d --name juice-shop \
  -p 127.0.0.1:3000:3000 \
  bkimminich/juice-shop:v20.2.0

Then I lost time to connection errors because I had configured both tools with localhost. Shannon starts its worker in Docker, where that address means the worker itself. Its route back to Ollama on the Mac had to be host.docker.internal. BugTraceAI ran from a Python virtual environment on the Mac, so its Ollama endpoint really did use localhost. Once I saw two network namespaces instead of one machine, the error was obvious.

BugTraceAI added one more bit of friction. Its custom-provider preset has sixteen model roles, and I had to point all of them at the same Qwen ID. Its pre-flight also wanted an API-key-shaped value even though Ollama ignores the key, so I used a local placeholder. The exact schema is in the custom-provider guide; copying the whole thing here would make this post age badly.

I also disabled BugTraceAI's screenshot validation after that path stalled. Qwen3.6 is multimodal, so this was not because the model cannot see images. It was simply the point in this particular integration where progress stopped.

Then thirteen agents met one model

Starting Shannon felt like the satisfying part. The target was reachable, the source was mounted read-only, Ollama was receiving requests, and the activity view started filling up. The system had made it past configuration and was actually doing something.

Then I counted the queue. Thirteen agent jobs had been queued against the same model endpoint. Ten minutes later, none had completed.

The model was not frozen. Ollama was processing requests; it was just doing so one at a time. That is its default for a loaded model, and increasing parallelism also increases the memory needed for every active context. A 64 GB machine sounds roomy until a 22 GB model, several long contexts, Docker and the rest of the toolchain all want their share.

So I stopped the run. Not because ten minutes proves Shannon is slow, and not because thirteen jobs describe some permanent part of its architecture. It only proved that this workload, this model and this configuration serialized badly on my machine.

That distinction matters. I had no finished scan, no findings to validate and no fair comparison between the two tools. Calling it a benchmark would turn an honest failed experiment into a much more confident story than the evidence supports.

Local is not the same as contained

The prompts and model responses stayed on my machine, and I did not need a billable API key. That was the part I wanted. But I caught myself mentally turning “local model” into “offline lab”, and those are not the same thing.

With Docker 28.1.1, binding Juice Shop to 127.0.0.1 kept it off the LAN; it did not block outbound traffic from Docker. A Python virtual environment kept BugTraceAI's packages separate; it did not sandbox the process. And Shannon reading a repository means the repository is model input, so untrusted source can carry prompt injection even when the mount is read-only. Shannon's own safety notes call this out.

What I would change next time

I would start smaller. One engine, one agent path and a short baseline request before launching the full workflow. I would record tokens per second, queue depth, memory pressure and time to the first completed task. Only then would I decide whether to use a smaller model, add parallelism or give the run more hardware.

I would also separate the questions. Keeping source and prompts away from a hosted provider is one goal. Saving money is another. Producing a useful pentest is a third. A local model can help with the first and remove the per-token invoice from the second, but this run says nothing yet about the third.

And I would still run the two tools. The white-box and black-box approaches are an interesting comparison, but only after each can finish on its own. The result worth comparing is not how busy their dashboards look; it is which findings survive a working proof.

Was it worth it?

Yes, just not for the result I expected. I did not come away with a list of Juice Shop bugs. I came away understanding where these systems put their concurrency, how quickly “local” becomes a networking problem, and why zero API spend does not mean zero cost.

The bill moved from tokens to hardware and waiting. That is still a useful trade if the point is privacy, learning or experimenting with the orchestration. On this setup, it was a bad trade if I needed a report by the end of the day.

Written 2 October 2026 · tell me I'm wrong