Jacob LagerrosDec 10, 2025
Securing AI Inference at Scale
Securing AI Inference at Scale
Jacob Lagerros
This is a demo of Waterwall: a retrofittable device that helps AI data centers protect model weights while serving massive amounts of inference, requiring no changes to inference engines and other hardware, and small compute overhead. Contact: jacob@ulyssean.com
Securing AI Inference at Scale
Jacob Lagerros
This is a demo of Waterwall: a retrofittable device that helps AI data centers protect model weights while serving massive amounts of inference, requiring no changes to inference engines and other hardware, and small compute overhead. Contact: jacob@ulyssean.com
Chapters
00:00
Waterwall Live Demo
This is a live demo of our FPGA-based AI security appliance, Waterwall.
01:03
Protecting AI models
02:31
"AGI Firewall"
02:55
Challenge 1: Verifying Inference
03:21
Challenge 2: Replay Non-Determinism
04:07
Challenge 3: Scaling Replay
05:59
Multi-GPU & Trillion-param Models
06:58
Summary
Transcript
00:00
Hi.
00:00
I'm Jacob, and this is a waterwall Let me show you something cool.
00:08
Here's an AI chat server, and I've inserted a water wall between the chat server and the model weights.
00:19
Now, you'll notice if I send a.
00:29
And I then go ahead and drag this slider, the speed of the chat slows down.
00:38
This is not a software effect.
00:41
I have made no edits to either the chat UI nor the inference stack.
00:46
Instead, as I'm changing this slider, I'm updating a hardware register.
00:53
I'm manipulating bits on the wire going through our device invisibly to either end of this connection.
01:03
Okay, but why?
01:08
Let me take a step back.
01:10
I think one of the most important problems of AI security is the following how can model developers serve inference to millions of users while, preventing attackers from stealing critical AIIP over that same high speed connection?
01:36
For example, if a sophisticated attacker gains access to our model server, they could hide in the noise of millions of API requests and exfiltrate something like our model weights in a manner of seconds.
01:59
Now, using a waterwall device, we can throttle the attack throughput in real time.
02:07
This slows down the attack, and it gives defenders a detection time not of minutes, but of hours, days or more.
02:16
And crucially, it does so without slowing down the speed of production inference.
02:28
Overall, you can think of this a bit like a firewall, but reimagined for the large AGI clusters of today, rather than the Internet of the 90s.
02:47
And to make this work, we have to solve three key problems.
02:55
First we need to identify genuine inference in a way that's extremely hard to spoof.
03:02
And to do so we introduce a replay server.
03:06
It simply replays inference on a trusted machine.
03:13
This produces a ground truth output that allows us to spot any deviations.
03:21
But second, how can we replay a prompt given randomness and non determinism in GPUs?
03:29
Well, it turns out this is a really solvable problem.
03:34
I'm running this live right now using a temperature setting that's non zero.
03:40
I am using a different GPU for the model server and the replay server.
03:44
There's an H100 and a Blackwell (RTX Pro 6000).
03:48
And nonetheless, as you'll notice, we end up achieving a replay score.
03:56
in this case, even 100%.
03:59
Sometimes better to get 98% during a demo because it looks more plausible.
04:04
but it's, yeah, it actually happened.
04:07
and now third, how on earth does this scale?
04:12
we obviously can't have one replay GPU for every model or like inference gpu.
04:19
Turns out it scales really well.
04:23
so if I go ahead here and submit a long prompt, Make sure to notice how long the model will spend on this generation.
04:37
As you'll see, due to the prefill / decode asymmetry of Inference, this entire 20 second generation can be replayed in just an instant.
04:50
so it's been going for a while as I've been talking.
04:52
and now get ready to watch the replay server.
05:02
There we go.
05:05
And this scales even further.
05:09
here.
05:10
I've gone ahead and recorded inference from lots of cloud GPUs so we can play back the chat not of one user but of 25 and responses not of 200 tokens but of 5,000 tokens.
05:25
So now we're going to have 25 GPUs for generation and still only a single GPU for our replay server.
05:34
So if we go ahead and fire this off, You'll notice that our replay server has no trouble keeping up.
05:48
It's cranking out more than 10,000 tokens per second at peak and maintaining similar accuracy levels in spite of the batching.
05:59
And just for completeness, in case you were wondering, all of this as well to multi GPU parallel settings.
06:11
So if I go ahead here and replace this with a 1 trillion parameter open source model.
06:20
So we need 8 GPUs to store just each instance of this model and I go ahead and fire off another test, we will still see ballpark similarly good accuracy and performance results.
06:43
And so, to sum up, the key problem we're solving is making it really hard to steal AI model weights while serving a ton of inference and spending only a negligible amount on compute overhead.
06:58
And this was a live demo, running end to end on a hardware and software installation in our lab.
07:05
And we're designing this to be a plug and play with any stack, GPUs, TPUs, or bespoke inference engines.
07:14
Our V1 prototype could be installed in a data center as soon as tomorrow, and we're aiming for, production deployments at the end of 2026.