Fine-tuning a certified network without losing its robustness guarantee

2026-11-17来源 dev.toEnglish

A certified classifier comes with a proof for each input: inside a radius r around the image, no perturbation can change the predicted class. In medical imaging that radius has to cover what happens in a clinic every day, such as a second photo of the same patch of skin taken at a slightly different angle.

The proof rests on the Lipschitz constant L of the network. The certificate divides the logit margin by L, so when fine-tuning doubles L, the radius at the same margin halves. Certified networks are trained on ImageNet while a clinic has a few thousand labelled images, so in practice you fine-tune, and in our experiments LoRA raised L by 1.9 to 2.5 times and full fine-tuning by 5 to 8.5 times.

A rotation keeps every singular value of the weights

The butterfly adapter multiplies each frozen weight matrix by an orthogonal matrix, W′ = R(θ)·W. For orthogonal R, (RW)ᵀ(RW) = WᵀW, so W′ has the singular values of W and its spectral norm stays where it was, whatever training does to θ.

R(θ) is a product of Givens rotations laid out as a butterfly: log₂ n levels, each of which pairs up the channels and rotates every pair by its own angle. A Givens rotation is orthogonal for any angle, so Adam trains the angles directly with no projection step or penalty term. Acting on unary states, the same rotation is a reconfigurable beam splitter (RBS) gate, which makes each adapter a quantum circuit of depth log₂ n. We simulate it on a GPU as an ordinary matrix.

A 2048 wide layer holds about 4.2 million weights, and its butterfly has 11 264 angles.

Check the bound in 20 lines

The layer ships in eigenkit, our open-source library (intro post). The script runs on a CPU in a few seconds.

pip install eigenkit==1.2.2 torch 
import torch from eigenkit.orthogonal.torch import Orthogonal, rotate_weight torch.manual_seed(0) n = 2048 W = torch.randn(n, n) / n**0.5 # a frozen pretrained weight L0 = torch.linalg.matrix_norm(W, ord=2) # its spectral norm (Lipschitz bound) # butterfly adapter: R(theta) @ W, R orthogonal for every theta R = Orthogonal(n) # butterfly layout of RBS / Givens gates with torch.no_grad(): for p in R.parameters(): p.normal_(0, 1.0) # any angles at all W_bfly = rotate_weight(R.matrix(), W) # LoRA, rank 4: W + B @ A, random factors standing in for trained ones A = torch.randn(4, n) * 0.05 B = torch.randn(n, 4) * 0.05 W_lora = W + B @ A print("butterfly angles:", sum(p.numel() for p in R.parameters())) print("LoRA params: ", A.numel() + B.numel()) print("L/L0 butterfly: ", f"{(torch.linalg.matrix_norm(W_bfly, ord=2) / L0).item():.3f}") print("L/L0 LoRA: ", f"{(torch.linalg.matrix_norm(W_lora, ord=2) / L0).item():.3f}") 
butterfly angles: 11264 LoRA params: 16384 L/L0 butterfly: 1.000 L/L0 LoRA: 2.764 

Any angles you draw give a ratio of 1 up to float32 rounding. The LoRA ratio depends on its factors: the random ones here only show the mechanism, and the trained LoRA adapters in the paper ended at 1.9 to 2.5.

Results on trichoscopy images from a clinic

With dermatologists at the University of Trieste we adapted BRONet, a certified ImageNet network, to trichoscopic scalp images for two tasks: diagnosis of hair loss across 5 classes (3 581 images from 143 patients) and activity of alopecia areata (1 196 images from 57 patients). The adapter acts on the 8 dense layers of BRONet's MLP, and every method trains a new classification head.

Diagnosis task, 717 test images:

Butterfly LoRA Full fine-tuning
Certified accuracy, ε = 36/255 63.8% 31.6% 35.6%
Certified accuracy, ε = 108/255 20.4% 0% 0%
Clean accuracy 80.4% 71.5% 82.5%
L / L₀ after adaptation 1.00 1.9 to 2.5 5 to 8.5
Trainable parameters about 100 000 141 000 33.6 million

Training only the new head also keeps L unchanged and reached 55.2% certified accuracy, so the butterfly adds 8.6 points by adapting the features as well. On the alopecia areata activity task the butterfly certified 40.6% of images, against 21.3% for LoRA and 17.9% for full fine-tuning.

We also attacked the adapted models with PGD and APGD-T. No attack found an adversarial example inside a certified radius, and the certified radius tracked the smallest successful attack on each image with a Spearman correlation of 0.97 to 0.99.

Paper and code

If you fine-tune a Lipschitz-constrained model on your own data, swapping LoRA for the butterfly takes two calls in the training loop: Orthogonal(n) for each frozen layer and rotate_weight(R.matrix(), W) in its forward pass.

原文:https://www.a8q9.cn/en/article/fine-tuning-a-certified-network-without-losing-its-robustness-guarantee-a9848b.html

相关文章