Can a learned bridge make video representations useful for language retrieval?
Current stateVideo-to-CLIP translation experiments with measured retrieval gains during development
Approach
The input is a set of videos with accompanying captions. Frozen V-JEPA 2 and CLIP encoders turn the frames and captions into cached feature arrays. V-JEPA supplies visual features; CLIP supplies image and text targets in a shared space. Training changes the translator between these representations while keeping the original encoders fixed.
The code explores linear, multilayer, residual, and transformer translators with cosine, mean-squared-error, and contrastive objectives. Later paths retain temporal tokens during training as well as supporting pooled video inputs. Retrieval then asks whether the translated video representation ranks the matching image or caption above other candidates. Configurations, checkpoints, and measurements feed a search that can use an LLM to propose subsequent trials.
Why this approach
A useful connection between spaces
The original purpose was cross-modal access: a visual observation should help recover relevant language-associated content. Matching vector dimensions or reducing an alignment loss is only part of that problem. The projected representation has to distinguish the right candidate from alternatives.
A second question is what the bridge leaves behind. Retrieval can improve even if information needed for motion distinctions or later cognitive tasks is lost. Those properties require their own comparisons; the retained results do not settle them.
Let later evidence revise the earlier conclusion
An early retrospective interpreted failed attempts as evidence that projection could not work. Subsequent experiments changed the objectives and architectures and reported positive retrieval. The evidence supports failure of those earlier training setups, not inherent incompatibility between the representation families.
Where it stands
The translator implementation includes several model families, training with temporal tokens, paired retrieval evaluation, checkpointing, and an LLM-guided configuration search. Its development summary reports best text Recall@1 of 37.1% and image Recall@5 of 94.4% on MSR-VTT; these are separate measurements, not one combined accuracy.
A separate experiment evaluated a residual translator trained toward a normalized mixture of averaged CLIP text and image targets. Training and evaluation code survive, including a small-pool retrieval evaluator. Reconstructing that exact run still requires its checkpoint, validation arrays, and a confirmed match between the recorded results and the evaluator version used.
What the work has shown
Later translators retrieved matching images and captions
The investigation continued beyond the earlier pooled linear bridge. The later implementation supports several translator families and 32 temporal tokens per video. Its development summary reports best text Recall@1 of 37.1% and image Recall@5 of 94.4% on MSR-VTT. These are separate reported bests; the aggregate record does not isolate which architecture change caused the gains.
The positive retrieval results revise the earlier conclusion that projection could not work. The remaining question is what the translated representation preserves for other uses, particularly motion and temporal distinctions. Those require measurements beyond retrieval scores.
Evaluating the translator against a mixture of image and text targets gave Recall@1 of 83% and Recall@5 of 96% in a 100-candidate evaluation. Across all 1,402 validation candidates, the recorded figures were 47.4% and 74.9%.
This run used a target combining 70% averaged text and 30% averaged image embeddings, followed by normalization. Validation also guided checkpoint selection. The numbers describe that development protocol; they are neither pure text-retrieval scores nor an untouched final-test estimate.
High cosine similarity did not guarantee retrieval
An early translator checkpoint reached mean cosine similarity of 0.86 but Recall@5 of only 24%. Its evaluator initially issued a passing verdict from cosine and stability while omitting the registered retrieval thresholds.
Later code ranks each projected video against candidate targets and reports Recall@1, @5, and @10 directly. These measures test whether the representation distinguishes the matching candidate from alternatives.
The simplest implemented translator averages video tokens when needed, applies a learned linear projection, and normalizes the output. It illustrates the attempted mechanism; it is not a trained checkpoint or an evaluation result.
Inspect the source excerpt
class LinearTranslator(nn.Module):
def __init__(self, input_dim: int, output_dim: int):
super().__init__()
self.linear = nn.Linear(input_dim, output_dim)
def forward(self, x: torch.Tensor) -> torch.Tensor:
if x.dim() == 3:
x = x.mean(dim=1)
return F.normalize(self.linear(x), dim=-1)