Rethinking the Readout: Unlocking Video Backbones for AI-Generated Video Detection

A natural choice for exploiting temporal signals is video backbones pretrained on large scale video datasets, since they are designed to model temporal dynamics across frames. Recent benchmarks, however, suggest the opposite. Several detectors built on image pretrained encoders have been shown to surpass video backbones with their default readouts on standard AIGV detection benchmarks. This suggests that the temporal advantage of video backbones is not being fully realized, and that the bottleneck likely lies elsewhere in the pipeline.

A standard video backbone takes an input clip and outputs a tensor of patch tokens of shape P × T × D, where P is the number of spatial patches per frame, T is the temporal length, and D is the channel dimension. These tokens carry rich spatiotemporal information, but downstream classification heads expect a single fixed length vector. The readout layer is what closes this gap. Common choices, such as global average pooling, a learnable CLS token, or fixed query attention pooling, differ in how the aggregation weights are computed but share the same operation, namely a joint spatiotemporal and semantic aggregation over patch tokens.

This kind of aggregation is the right design for tasks that rely on a stable global understanding, such as action recognition, where local frame to frame fluctuations and inter patch differences are noise to be averaged out. AIGV detection has the opposite need. The discriminative signal between real and generated videos lives precisely in those small frame to frame variations within each patch and in the relations among patches. Standard readouts smooth out the former and dissolve the latter in a single step, leaving the pooled vector blind to the very cues that AIGV detection most reliably depends on. We formalize this mismatch as two complementary losses, namely the loss of inter patch relations and the dilution of per channel velocity magnitudes.

To make this concrete, we focus on GAP as the simplest case and base the analysis on the patch velocity

where z(p)t ∈ ℝ^D denotes the p-th patch token of frame t. Applying GAP across patches and then taking a temporal difference is equivalent to averaging the patch velocities,

The patch axis collapses from P × D to 1 × D in a single step, and an analogous collapse happens along the temporal axis. This collapse erases inter patch relations and cancels velocity components across channels, the two effects we analyze below.

The two losses share the same source. The standard readout is designed for joint spatiotemporal and semantic aggregation, which by its very design erases inter patch structure and dilutes channel level magnitudes. Recovering both requires intervening at the readout rather than within the backbone.

Figure 3 shows the overall design of V-PVP. We keep the backbone fully frozen and replace only the readout with two parallel streams over the patch velocity field {v(p)t}. A velocity gated learnable aggregation z̃_t addresses Loss 1 by carrying inter patch heterogeneity into the pooled output, and a channel faithful velocity stream r_t addresses Loss 2 by preserving per channel magnitudes through an element wise nonlinearity. The two streams are projected independently, concatenated at each time step, and passed to a shallow temporal head.

Let z(p)t be the p-th patch token of frame t and v(p)t = z(p)t+1 − z(p)t the patch velocity. A small scoring network produces a per patch score on the fly, and a softmax converts scores into aggregation weights,

Standard readouts collapse the patch dimension with weights that are fixed once training ends, applying the same weighting regardless of which patches are anomalous. z̃_t differs in two respects. First, the learned α(p)t can vary by orders of magnitude across patches within a single frame, whereas the uniform 1/P of GAP dilutes every patch equally. Second, α(p)t is computed from the current video's (z(p)t, ‖v(p)t‖₂), so the spatial distribution of weights itself encodes inter patch heterogeneity. This weight pattern takes different shapes across videos and propagates into the channel pattern of z̃_t through the weighted sum. In effect, the P-dimensional weight vector α_t acts as a per video carrier of the inter patch structure that motivated Loss 1, allowing z̃_t to retain a discriminative signature of this structure even after collapsing the patch axis.

The scoring network receives ‖v(p)t‖₂ explicitly alongside z(p)t. Scoring on appearance alone reduces the aggregator to a semantic selector that overfits to the spatial fingerprint of the training generator (Table 3b). Conditioning on velocity magnitude anchors the definition of “anomalous” to a generator invariant physical quantity and prevents fingerprint memorization.

Although z̃_t addresses Loss 1 by introducing input-adaptive patch weighting, it remains a signed linear aggregation over patch-level signals. Velocity components from different patches can still cancel on each channel when they have opposite signs, so a signed linear readout cannot guarantee faithful preservation of per-channel velocity magnitudes. Loss 2 therefore requires a nonlinear operation before spatial aggregation.

The r_t stream avoids this by applying an element wise absolute value before the spatial sum,

Moving the nonlinearity inside the sum prevents opposite signed components from cancelling, so per channel velocity magnitude is faithfully preserved. The resulting vector encodes both how much anomalous activity exists and which channels it occupies.

We use the mean rather than the max because channel level anomalies in generated videos typically appear as weak signals distributed across many patches. The mean accumulates these diffuse contributions and is robust to isolated noise.

At each frame pair, z̃_t and r_t each produce a D-dimensional vector with very different statistics. z̃_t inherits the near zero mean signed distribution of post LayerNorm features, while r_t is strictly non-negative and typically an order of magnitude smaller. A shared projection cannot rescale both. We apply two independent linear maps to H/2 each and concatenate along the channel axis at each time step,

Splitting the channel budget keeps the two signals on a common scale before fusion, and bounds the readout at the 0.5M parameter level.

A short temporal convolutional head reads {x_t} into a single logit. Because u_t and w_t share the channel axis, every kernel position sees both signals at the same time step. A kernel size of three extends this to consecutive frame pairs, which matches the temporal scale of the inter frame inconsistencies V-PVP targets. A single anomalous transition is often not informative on its own, but a short window of them is. We then mean pool over time rather than learn a temporal attention, to avoid overfitting to the training generator's temporal position patterns.

The aggregator, the projections, and the head are trained from scratch. The backbone remains fully frozen.

Datasets and protocol. We evaluate V-PVP on two cross-generator benchmarks. AIGVDBench contains a split spanning 20 generators across T2V, I2V, and V2V tasks. We adopt its three-generator training protocol: OpenSora, CogVideoX1.5, and EasyAnimate serve as fake training sources. GenVidBench-143k adopts a cross-source protocol in which the training sources (Pika, VideoCrafterV2, ModelScope, T2V-Zero) and the test sources (MuseV, SVD, CogVideo, Mora) are fully disjoint. The primary metric is the mean per-generator AUC. Frames are sampled and preprocessed following the AIGVDBench protocol. The default backbone is VideoMAE pretrained on Kinetics-400 and kept fully frozen, with patch tokens extracted from the pre-final-norm position of the last transformer block.

Baselines. We compare V-PVP against three families of baselines under the benchmark-specific cross-generator protocols. AI-generated image detection models classify each frame independently and aggregate by majority vote. AI-generated video detection models are designed specifically for AIGV detection and use their original temporal modeling heads. Video classification models are adapted by full-parameter fine-tuning on the same training set, so that the comparison reflects each backbone's best achievable performance. All baselines are retrained on the corresponding training split following the unified evaluation protocol of each benchmark. Each method retains its original architecture and the hyperparameters recommended by the original authors, and only the training data is replaced.

Tables 1 and 2 report results on AIGVDBench and GenVidBench-143k. V-PVP reaches mga AUC of 95.28 on AIGVDBench and 93.75 on GenVidBench-143k, ranking first on both benchmarks. The backbone stays fully frozen and only about 0.5M parameters are trained.

Under the standard readout, some fully fine tuned video backbone in Table 1 sits below the best image level baseline, indicating that releasing backbone parameters alone does not unlock the temporal advantage promised by video pretraining. Our frozen backbone V-PVP exceeds end to end VideoMAE fine tuning (about 87M trainable parameters) with the standard readout by +6.94 AUC, and also surpasses the strongest image level baseline. With the backbone kept fully frozen, V-PVP simultaneously outperforms all video classification baselines, which are trained with full fine-tuning under their original readouts, as well as all image pretrained detectors, while training roughly two orders of magnitude fewer parameters. This suggests that the standard aggregation readout is an important bottleneck for video backbones in AIGV detection.

To test whether the bottleneck is the readout rather than the representation, we port V-PVP to several frozen video backbones and compare against the standard readout and adaptation alternatives.

We cover four ViT-based encoders with diverse pretraining objectives: TimeSformer, VideoMAE, UniFormerV2, and the multimodal large model Qwen3-VL-8B. On every backbone we compare two groups of methods. The first keeps the backbone frozen and only trains a head, including a linear probe, a matched-size head, and our V-PVP. The second updates backbone parameters, including LoRA fine-tuning with rank 8 and full fine-tuning. Full fine-tuning is omitted on Qwen3-VL-8B due to its ~8B parameter scale.

Figure 4 reports mga AUC on the AIGVDBench open-source split. V-PVP achieves the highest AUC on every backbone, surpassing all readout and adaptation alternatives while training roughly two orders of magnitude fewer parameters than full fine-tuning. The advantage holds across all four pretraining objectives, which supports the readout-bottleneck hypothesis: the bulk of the fine-tuning gain comes from adjusting how patch tokens are aggregated, not from modifying the representation itself. We also note that backbone tuning does not always help. On UniFormerV2, both LoRA and full fine-tuning stay below 80, lower than the matched-size head with the backbone frozen. This suggests that updating backbone parameters can amplify overfitting to the training generator's spatial fingerprint rather than help. V-PVP avoids this failure mode by anchoring the readout to a generator-invariant quantity and reaches 93.56 on the same backbone.

A natural follow-up question is whether the gains of V-PVP depend on a frozen backbone. We test this on VideoMAE by pairing V-PVP with LoRA at rank 8 and with full fine-tuning, and compare against the same adaptation schemes under the standard readout. Table 4 reports mga AUC on AIGVDBench. The standard-readout column is nearly flat across adaptation regimes, indicating that backbone capacity has little outlet once patch tokens are collapsed by joint spatiotemporal pooling. V-PVP responds in the opposite way and improves over the standard readout under every adaptation scheme, with the largest margin reaching 8.9 points under full fine-tuning. Even the frozen configuration with V-PVP, at roughly 0.5M trainable parameters, surpasses full fine-tuning under the standard readout at roughly 87M parameters. The right axis along which to invest capacity is therefore the readout rather than the backbone.

We ablate V-PVP along two axes. The first looks inside V-PVP and asks how much each stream and the velocity gating contribute. The second looks outside and asks whether other readouts that drop the velocity prior can reach the same accuracy. All ablation runs use the frozen VideoMAE-K400 backbone with head-only training. All ablation rows including V-PVP use a fixed 5-epoch budget for fair comparison. Results are reported as mean AUC ± standard deviation over four random seeds.

Per-stream contribution

Table 3a dissects V-PVP into its two streams. Going from r-only (MLPMean, 79.10) to z̄ + r adds +11.13, most of which comes from the semantic information carried by the GAP pathway. Replacing z̄ with the velocity-gated z̃ adds another +4.11 on top of an already channel-faithful stream. The middle two rows show that both components are needed: removing the r stream drops AUC by 1.27, and removing the velocity input from the scoring network drops it by 2.26.

Alternative patch-level readouts

A natural question is whether a free-form head that keeps the patch dimension but drops the velocity prior could match V-PVP. Convolutional heads (Conv2D, Conv3D) and a spatial-attention head are applied to the per-patch energy map E(p)t = ‖v(p)t‖₂, and a PatchMLP consumes the raw patch tokens directly as the most expressive alternative. Heads with more capacity to use the spatial dimension fit the training set more aggressively but generalise worse: PatchMLP and Conv2D reach near-perfect train AUC, yet their test mga AUC stays well below V-PVP. Without an explicit prior, the spatial pattern that supervision finds is generator-specific and does not transfer. V-PVP avoids this by anchoring patch selection to the generator-invariant quantity ‖v(p)t‖₂, outperforming Conv2D by 25.73 points (94.34 vs. 68.61).

We identify the readout layer as the principal bottleneck that prevents video pretrained backbones from realizing their temporal advantage on AIGV detection. The standard CLS or GAP aggregator performs a joint spatiotemporal and semantic collapse over patch tokens, which erases inter patch relations and dilutes per channel velocity magnitudes. Both losses remove the cues that AIGV detection most reliably depends on.

Building on this diagnosis, we propose V-PVP, a lightweight readout that keeps the backbone fully frozen and replaces only the aggregation layer with two parallel streams over the patch velocity field. The velocity gated aggregation z̃_t conditions patch attention on ‖v(p)t‖₂ to carry inter patch heterogeneity into the pooled output, and the channel faithful stream r_t preserves per channel magnitudes through an element wise nonlinearity. V-PVP adds only about 0.5M trainable parameters. Across multiple ViT based video backbones and two cross generator benchmarks, V-PVP consistently surpasses same backbone readouts and exceeds end to end fine tuning with the standard readout. With VideoMAE on AIGVDBench, our method reaches 95.28 mga AUC while keeping the backbone fully frozen, and even outperforms full fine tuning under the standard readout. These results indicate that for AIGV detection, the right axis to invest capacity is the readout rather than the backbone.

Limitations

V-PVP has several known limitations. First, generators that deliberately mimic real video statistics, for example by injecting inter frame jitter or preserving high frequency spatial detail, can shrink the inter patch separability of ‖v(p)t‖₂ and drive velocity gating toward uniform attention. Second, very short clips with T < 4 leave too few temporal samples of patch velocity and increase the variance of z̃_t and r_t. Third, when the backbone has not been pretrained on a distribution close to the target domain, the frozen patch tokens may be too weak to support readout only learning, and LoRA or full fine tuning would be more appropriate. Finally, the readout bottleneck still exists on CNN based backbones but in a softened form, since internal convolutions already perform part of the spatial aggregation before the readout, partially absorbing the patch level evidence that V-PVP relies on. The paper's appendix confirms this on I3D-R50 and SlowFast-R50, but a thorough study of CNN specific readout design is left to future work.