Type a section name and press Enter. Use “help” for the list.
Earlier architecture · 3–4 October 2026
Research archive
The earlier text and caption experiments retain their results and limitations. They explain the route here; they do not establish what the current visual substrate can do.
The 7 October reset made a continuing perceptual life and open-ended acquired content the organizing requirements. The subsequent visual studies live in the current run gallery.
01 · Memory and replay
Remembering what came next
Four reader runs used Alice in Wonderland, reduced to 108,082 letters. Three read the original order with signal-selected replay, uniform replay, or no replay. The fourth experienced the book with its words shuffled.
The measure asks which unit came next at 500 positions in the final quarter of each target sequence. The ordered readers use their own reading logs for cues and targets. The shuffled reader is tested on the original order, which it did not experience. Each run develops its own units.
| Condition | 1-unit cue | 2-unit cue | 4-unit cue | 8-unit cue |
|---|---|---|---|---|
| Signal-selected replay | 5.8% | 23.6% | 38.2% | 54.8% |
| Uniform replay | 9.4% | 20.6% | 38.2% | 57.6% |
| Shuffled-word life | 3.6% | 11.6% | 8.4% | 3.6% |
| No replay (letters) | 10.4% | 8.8% | 7.4% | 10.0% |
| Unit-pair baseline (signal arm) | 15.6% | 14.0% | 12.2% | 11.0% |
The unit-pair baseline counts which unit most often follows the final cue unit in the same sequence. Learned units need not be whole words.
| Life / reference | Units known | Boundary F1 |
|---|---|---|
| Signal-selected replay | 921 | 0.616 |
| Uniform replay | 677 | 0.651 |
| Shuffled words | 669 | 0.631 |
| No replay | 26 | 0.397 |
| Exact-count reference | — | 0.693 |
| Random boundaries, same count as reference | — | 0.241 |
What I take from it
The replay readers formed multi-letter units and retrieved parts of the experienced sequence. The no-replay run retained only its 26 letter units. The shuffled reader also formed units, but its next-unit recall was low when tested on the original order.
What remains unresolved
This does not establish that replay beats an online learner receiving equivalent information. Uniform replay scores higher at eight-unit cues; signal-selected replay scores higher at two. One seed cannot establish a reliable difference. A system with prior experience also showed interference when later recalling video; additional experience did not consistently improve recall.
02 · Grounded word learning
Learning word associations from caption context
For each of 100 probe words, the association-learning stage withholds photos whose captions contain that word. Other words acquire associations from the remaining captioned scenes. The held-out word then receives combinations of those grounded associations through text contexts.
This withholding applies only to word-association learning. The visual encoder was trained on the COCO images before the per-word holdouts. Word associations are accumulated without a pretrained language model or supplied semantic word embeddings. The starting pool contains 118,287 COCO photos and 591,753 captions. Word boundaries are supplied in this experiment. Neighbors are weighted by positive pointwise mutual information (PPMI), a measure of their co-occurrence strength.
Word lift is precision among the ten highest-ranked test images, divided by the fraction of test images whose captions contain the word. A lift of 1 is the base rate. This uses caption occurrence as a proxy for visual relevance.
| Visual seed | Direct experience | Context only | Scrambled control | Context − control | 95% interval of difference |
|---|---|---|---|---|---|
| Seed 1 | 3.181 | 2.206 | 0.930 | +1.276 | 0.761 to 1.847 |
| Seed 2 | 3.026 | 2.285 | 1.105 | +1.179 | 0.622 to 1.795 |
| Seed 3 | 3.330 | 2.551 | 1.215 | +1.337 | 0.766 to 1.899 |
What I take from it
Context-based associations have higher mean word lift than the corrected scrambled control in all three seeds. In this setup, caption co-occurrence provides information for retrieving images associated with words that were withheld from direct word–photo pairing.
What remains unresolved
Flickr8k’s test split has been used repeatedly during development. These results need confirmation on fresh data. The measure does not establish general understanding, word order, or who did what to whom. Nor does accumulation and similarity by itself establish an advantage specific to VSA.
A correction that changed this experiment
An earlier letter-string version allowed shorter words inside a held-out word to leak information: “snow” inside “snowy,” for example. The reported experiment uses whole words. The first PPMI control also retained the real weights; it was withdrawn and replaced by the corrected control reported here.
Inspect all 100 words across the three seeds
100 of 100 words shown · Alphabetical order · Lift values as recorded
| Word | Direct lift | Context lift | Control lift | Photos withheld | Captions read |
|---|---|---|---|---|---|
| across | 0.00 | 0.00 | 0.00 | 3,102 | 3,572 |
| air | 5.05 | 5.05 | 0.00 | 4,772 | 7,425 |
| along | 1.67 | 0.00 | 1.67 | 3,718 | 4,106 |
| another | 1.27 | 3.80 | 0.00 | 4,077 | 4,879 |
| area | 2.56 | 2.56 | 0.00 | 7,880 | 8,681 |
| around | 1.59 | 0.00 | 0.00 | 6,801 | 8,444 |
| background | 1.52 | 0.00 | 1.52 | 5,304 | 5,792 |
| ball | 6.85 | 5.48 | 1.37 | 5,125 | 11,181 |
| beach | 7.94 | 3.17 | 4.76 | 3,872 | 11,211 |
| bicycle | 0.00 | 0.00 | 4.35 | 1,204 | 2,276 |
| bike | 2.27 | 0.00 | 0.00 | 2,044 | 3,788 |
| black | 3.11 | 0.89 | 0.44 | 12,895 | 19,057 |
| blue | 2.33 | 0.58 | 1.16 | 8,916 | 13,537 |
| boy | 2.37 | 2.37 | 0.59 | 5,158 | 10,776 |
| boys | 5.00 | 2.50 | 0.00 | 1,083 | 1,708 |
| brown | 2.04 | 2.72 | 1.36 | 6,262 | 8,263 |
| building | 6.98 | 4.65 | 0.00 | 6,864 | 12,326 |
| camera | 0.00 | 0.00 | 0.00 | 2,623 | 3,300 |
| child | 2.31 | 0.77 | 0.00 | 3,723 | 6,074 |
| children | 0.00 | 2.99 | 0.00 | 1,816 | 3,261 |
| covered | 3.23 | 3.23 | 0.00 | 5,953 | 7,070 |
| crowd | 9.09 | 3.64 | 0.00 | 2,211 | 2,851 |
| dirt | 4.17 | 2.08 | 0.00 | 2,636 | 3,919 |
| dog | 0.92 | 0.46 | 0.92 | 4,708 | 17,810 |
| dogs | 0.00 | 0.00 | 0.00 | 994 | 2,925 |
| doing | 2.13 | 2.13 | 6.38 | 2,347 | 3,279 |
| dressed | 1.72 | 0.00 | 1.72 | 1,735 | 2,144 |
| each | 0.00 | 2.63 | 0.00 | 5,641 | 6,950 |
| field | 7.79 | 5.19 | 0.00 | 11,276 | 20,748 |
| girl | 3.97 | 3.31 | 1.99 | 4,673 | 9,315 |
| girls | 2.08 | 2.08 | 0.00 | 925 | 1,626 |
| grass | 3.12 | 3.12 | 0.00 | 7,666 | 12,985 |
| grassy | 9.30 | 6.98 | 0.00 | 3,533 | 4,323 |
| green | 1.10 | 0.00 | 0.00 | 8,651 | 12,455 |
| ground | 2.78 | 0.00 | 0.00 | 3,667 | 4,433 |
| group | 3.37 | 0.00 | 1.12 | 13,261 | 21,509 |
| hat | 2.17 | 2.17 | 2.17 | 1,839 | 3,370 |
| high | 0.00 | 0.00 | 0.00 | 1,917 | 2,270 |
| hill | 7.14 | 10.71 | 0.00 | 2,781 | 3,963 |
| him | 2.00 | 2.00 | 0.00 | 1,848 | 2,005 |
| holding | 1.75 | 0.00 | 0.00 | 15,587 | 27,820 |
| holds | 0.00 | 2.13 | 0.00 | 3,062 | 3,330 |
| jacket | 5.77 | 3.85 | 3.85 | 1,505 | 1,780 |
| jump | 2.78 | 2.78 | 0.00 | 808 | 996 |
| jumping | 0.93 | 0.93 | 0.93 | 1,913 | 2,994 |
| jumps | 0.00 | 1.25 | 0.00 | 569 | 623 |
| kids | 0.00 | 0.00 | 0.00 | 1,397 | 1,860 |
| large | 1.82 | 0.91 | 0.00 | 18,482 | 24,138 |
| little | 1.23 | 1.85 | 0.00 | 4,692 | 6,412 |
| long | 3.03 | 3.03 | 0.00 | 2,928 | 3,555 |
| looking | 1.43 | 4.29 | 0.00 | 9,293 | 11,899 |
| looks | 0.00 | 1.75 | 1.75 | 2,899 | 3,033 |
| man | 1.91 | 1.91 | 0.64 | 26,675 | 72,297 |
| men | 3.33 | 5.00 | 1.67 | 5,483 | 10,618 |
| mouth | 4.17 | 0.00 | 0.00 | 1,188 | 2,082 |
| ocean | 7.69 | 7.69 | 0.00 | 3,320 | 5,357 |
| orange | 11.11 | 0.00 | 0.00 | 3,050 | 4,860 |
| outside | 1.54 | 0.00 | 1.54 | 6,572 | 7,782 |
| park | 4.88 | 0.00 | 0.00 | 4,334 | 6,809 |
| path | 4.00 | 0.00 | 0.00 | 920 | 1,121 |
| people | 1.89 | 1.26 | 0.00 | 18,404 | 41,630 |
| person | 0.79 | 0.79 | 2.36 | 15,855 | 24,505 |
| picture | 0.00 | 2.78 | 0.00 | 7,391 | 8,792 |
| pink | 9.30 | 2.33 | 0.00 | 1,965 | 3,241 |
| play | 4.48 | 4.48 | 4.48 | 1,595 | 1,792 |
| player | 2.86 | 2.86 | 0.00 | 4,834 | 9,670 |
| playing | 3.55 | 3.55 | 3.55 | 8,075 | 14,657 |
| plays | 3.45 | 1.72 | 0.00 | 745 | 798 |
| red | 3.73 | 1.86 | 1.24 | 10,330 | 17,086 |
| rides | 6.25 | 2.08 | 2.08 | 1,767 | 1,981 |
| riding | 3.33 | 0.00 | 0.00 | 10,089 | 18,027 |
| run | 0.00 | 0.00 | 4.00 | 699 | 755 |
| running | 2.75 | 1.83 | 1.83 | 1,485 | 2,248 |
| runs | 7.84 | 3.92 | 1.96 | 320 | 357 |
| shirt | 0.00 | 0.74 | 1.47 | 4,070 | 5,154 |
| side | 3.45 | 3.45 | 0.00 | 9,392 | 11,406 |
| sidewalk | 7.41 | 11.11 | 3.70 | 3,339 | 4,859 |
| sits | 4.44 | 0.00 | 0.00 | 8,255 | 9,236 |
| sitting | 4.65 | 3.49 | 0.00 | 31,402 | 52,737 |
| small | 2.31 | 2.31 | 0.00 | 15,158 | 19,717 |
| snow | 12.07 | 8.62 | 0.00 | 4,914 | 11,369 |
| snowy | 17.95 | 12.82 | 0.00 | 2,490 | 3,460 |
| something | 0.00 | 0.00 | 2.63 | 2,243 | 2,476 |
| stand | 0.00 | 0.00 | 2.50 | 4,065 | 4,627 |
| standing | 0.65 | 0.65 | 1.94 | 25,367 | 42,397 |
| stands | 1.10 | 0.00 | 2.20 | 4,973 | 5,489 |
| street | 6.06 | 0.00 | 0.00 | 13,231 | 29,056 |
| through | 2.74 | 2.05 | 0.68 | 6,869 | 8,606 |
| top | 0.00 | 0.00 | 0.00 | 17,860 | 22,049 |
| trick | 3.33 | 3.33 | 3.33 | 1,901 | 2,994 |
| walk | 4.17 | 0.00 | 4.17 | 2,093 | 2,360 |
| walking | 1.14 | 1.14 | 1.14 | 7,953 | 14,532 |
| walks | 2.04 | 2.04 | 2.04 | 1,466 | 1,693 |
| water | 2.14 | 2.14 | 0.00 | 8,054 | 16,247 |
| wearing | 1.89 | 2.26 | 0.00 | 7,114 | 9,884 |
| white | 2.37 | 0.79 | 1.19 | 23,991 | 35,148 |
| woman | 2.82 | 2.11 | 0.00 | 12,877 | 33,904 |
| women | 0.00 | 5.71 | 0.00 | 2,962 | 4,903 |
| yellow | 5.00 | 1.25 | 1.25 | 5,181 | 8,315 |
| young | 1.82 | 0.45 | 1.36 | 12,216 | 16,982 |
Photos are withheld from word-association learning after visual training. The full download retains evidence counts and the additional wrong-weight control.
03 · Returning the mechanism to the full system
After the same 6,000 photos
The word-learning spike and Cliste’s ongoing memory system are different experimental settings. This comparison brings the image-trained encoder into Cliste, then checks its own learned units and beliefs against a simple centered visual prototype built from exactly the same 6,000 Flickr8k photos and captions.
Both sides are measured immediately after the photos, before further video or reading. They share the visual encoder, 1,000 evaluation photos, 4,995 captions, and scoring procedure. The comparison uses the 98 probe words both sides represent.
| Measure | Cliste | Matched reference |
|---|---|---|
| Probe words measured / 100 | 98 | 100 |
| Mean word lift, 98 shared words | 4.051 | 4.031 |
| Shared words above 2× base rate | 70 | 72 |
| Caption → photo recall@10 | 46.3% | 48.1% |
Paired mean word-lift difference: +0.020.
95% bootstrap interval: -0.127 to 0.176.
There is no clear word-lift advantage in this run. The interval includes zero; the experiment does not establish superiority or equivalence. The reference additionally represents “area” and “path,” which are excluded from the paired word comparison.
Recall@10 here ranks the correct photo among 100 candidates, with expected credit for ties. It is not full-gallery Flickr8k retrieval. These numbers also should not be treated as a learning curve continuing the COCO study above: the experience budgets differ.