Type a section name and press Enter. Use “help” for the list.
Figures: softened for dark viewingDisplay only; image files are unchanged.
Cliste → Sophia · A continuing investigation
Learning how to build a mind from experience.
I’m developing Cliste as a cognitive substrate for Sophia, guided by how human brains perceive, remember, relate experiences and learn.
My starting position is that text is not the substrate of cognition or intelligence. Language participates in thought, but perception, memory and action need machinery that can operate before an experience has been described in words.
The brain’s roughly 20-watt metabolic budget motivates the search for more economical ways to represent and use experience.
How I arrived at vector symbolic architectures
The question came before the representation: how could Sophia build a model of the world from experience, remember what happened, and use it in later thought? In LOGOS, I pursued that through a Hybrid Cognitive Graph (HCG). It combined explicit graph structure with vector embeddings: entities and their relationships could be recorded and traversed, while vector similarity helped locate related content. Language was meant to be an interface to that cognitive core.
Joint Embedding Predictive Architecture (JEPA) was another major influence. Its approach to learning through predictions of representations suggested a route to grounding in perception, without requiring every experience to arrive as a sentence. I explored video representations and their connection to language-associated representations in the JEPA translation experiments. The longer-term idea was to give the graph perceptual evidence alongside its language-derived content; that was a research direction, not an accomplished integration.
The graph work exposed a problem with my expectations of embeddings. I had treated proximity in a learned vector space as more authoritative about relationships than it deserved to be. Two things can be close because they occur in similar descriptions, without one being part of the other or causing it. I tried giving graph structure a larger role, too. For Sophia, I eventually rejected that graph-plus-embedding design as the cognitive foundation: finding associated information was not enough to make its contents usable in flexible thought.
I hadn’t started out knowing about Vector Symbolic Architectures (VSA). Researching alternatives led me to this established field and to a different role for vectors. Similarity remains useful, but there is also an algebra for constructing representations: bind content to a role, bundle several qualified contents together, and use the roles to recover approximate constituents. The vector structure itself participates directly in these compositional operations.
That is what made composition so compelling. An image can be bound to where and when it was encountered, and those qualified contents can be combined into an experience. A tag needs to modify the intended representation; adding an unrelated vector beside it would not preserve that attachment. This offers a way to connect perception, memory and internal construction through a common set of operations.
I was already reaching for a mind built from reusable representations and pointers to experience. Chris Eliasmith’s How to Build a Brain and the Semantic Pointer Architecture (SPA) gave me a more developed framework and concrete mechanisms for that direction. Here, SPA means a cognitive architecture using structured vector representations across neural systems. Binding, cleanup toward familiar patterns, and cooperation among specialized systems became practical questions for Cliste.
Cliste now investigates a shared hyperdimensional medium with specialized neural machinery for perception and other operations. JEPA remains an influence on learning from perception; VSA provides the compositional operations being tested. Neither the algebra nor the architectural inspiration supplies the acquired meanings, useful relationships or procedures. Those still have to be learned and demonstrated in a continuing system.
What the current system does
Natural images enter at their native dimensions. A learned encoder produces regional features, which are composed into VSA memories. Later visual fragments can query those memories, and a decoder renders the retained content. Original image files are unavailable to the isolated recall workers.
No captions, text queries or LLM interpret this visual-memory path.Neural encoders and decoders connect pixels and vectors. Visibility masks, spatial coordinates, clocks and experimental requests are currently supplied.
The life keeps encountered content, temporal context, and current working content distinct. Recall can bring an earlier experience into working memory without recording it as a new perception, and held working content can survive new intake and restart. The encoder and decoder provide the current pixel-to-vector interface; a separate acquired pattern bank is being tested for cleanup and reuse.
01See
Pixels become perceptual features.
02Remember
VSA stores content and structured context.
03Recall
A visual cue selects retained content.
04Inspect
The decoder renders the selected memory.
Cliste had stored this photograph earlier; the fragment retrieves that memory. The right panel is decoded from the stored vectors.Complete experiment →
Results and interpretation
Each experiment links to its methods, measurements, counterexamples and original verdict.
A visual fragment can open a remembered sequence
A visual-memory experiment connects the pieces in one continuing life. A visible fragment recalls an earlier image, then retained temporal context selects the next two encounters. Two local views of each of 32 photographs were tested at two partial visibility levels. The support-adjusted policy completed 60 of those 128 paths, compared with 25 for the baseline. No wrong path occurred in that stored-photo comparison; the rest abstained.
One remembered step survived another 24 encounters and a save/reload before replay resumed. Context-swap controls changed which memories followed, supporting a role for the retained associations. Quality remains provisional: earlier unknown-source false accepts, competition from new memories, and weak video recall remain in the record. These are remembered encounters, not predictions of unseen events.
Retaining reconstructed regional vectors on the GPU reduced median recall-query time from 4.154 to 1.022 seconds, a 75.4% reduction once the cache was populated. The comparison used 64 cue fields from 16 photographs, repeated three times against a history of 135 encounters. It measures cue encoding, memory search and recall application; decoding and remembered continuation were checked outside the timer.
Recall-query cost on one GPU and memory history
Measurement
Cache disabled
Cache already filled
Median query time
4.154 s
1.022 s
95th-percentile query time
4.679 s
1.109 s
Temporary GPU cache
0
2.99 GiB
All 384 paired decisions matched, including the existing false recall and unresolved queries. Checks of recalled content, temporal continuation, new admissions and saved state also passed. This reduces repeated computation without improving recall quality or shrinking durable storage. The first empty-cache query took 4.588 seconds; the cache remains optional.
Memories can be revisited through temporal context
A temporal experiment produced 276/276 correct earlier/later decisions across 46 anchors, two directions and three conditions, including six correct stops at the history’s boundaries. Permuting the storage rows preserved the selections; swapping temporal contexts changed them with the new associations.
Those controls support a specific conclusion: the retained temporal representation was selecting the content, rather than its row position acting as a hidden chronology. Replay also resumed exactly across a save and restart. These are repeated mechanism checks in one history, not 276 independent lives or a demonstration of prediction.
The same algebra can bind acquired content to spatial and temporal context. The current algebra tests supplied coordinates and reversible bindings; learned spatial understanding and moving visible objects in the current representation remain open.
The starting point is the already-held memory 38. A programmed request advances through the history; the first three of seven steps return 39, 40 and 41. Each matches the next encounter in the observer’s log, which was unavailable to the selector. The pictures are decoded from retained vectors. The 276-decision figure describes the separate full set of earlier/later controls.Temporal controls, replay images and qualifications →
Compact vectors can preserve useful recall
Eight-bit dense storage reduced the measured tensor payload from 616.75 to 176.71 MiB (71.35%), preserving all 192 matched retrieval choices in a development comparison. The shared model’s cost is separate. This was numerical compression; it does not establish that the acquired pattern vocabulary is saving space.
Fidelity requirements should follow downstream use: preserve the distinctions needed for recall, composition and later action. Testing those uses can establish where approximation is acceptable and where lost detail changes the outcome.
In a partial-image experiment, the correct stored image ranked first for 184 of 192 cues. The original acceptance rule used only 66 of those cues. Accounting for how much visual evidence was visible raised correct accepted recalls to 111, with the same encoder, decoder and rankings.
Partial visual recall: 32 stored photographs, six related cues per photograph
Decision
Original rule
Support-adjusted rule
Correct recall
66 / 192
111 / 192
Wrong recall of a stored-source cue
0 / 192
0 / 192
No recall accepted
126 / 192
81 / 192
False match: 64 never-stored photographs
5 / 384
7 / 384
False match: first panel before storage
2 / 192
10 / 192
The gain exposes usable information already in the representation. It does not mean the visual encoder learned more. The extra false matches show why accepting more candidates is not sufficient: a plausible fragment can belong to the wrong experience.
The adjusted rule failed its registered introduced-error criterion: a 9.51% upper confidence bound exceeded the 5% target. I’m retaining it as a provisional comparison for downstream work, with that failure intact. The next question is whether additional perceptual evidence can resolve a tentative match before the system relies on it.
Inspect a false-match exampleThe adjusted rule accepts a sky fragment as evidence for the wrong memory. The cue comes from a kite photograph that was never stored; the selected memory is an aircraft scene.Methods, all results and uncertainty →
Replay can preserve old memories while perception improves
Further training improved reconstruction of fresh photographs. Ordinary training also damaged decoding of already stored vectors. Rehearsing retained vectors during training largely preserved their readback, at a cost to the improvement on fresh images.
Native continuation: relative changes in mean pixel error; three paired training seeds
Measurement
Ordinary training
Training with replay
Fresh Flickr photos (256 images)
10.11% lower
5.28% lower
Stored, unrehearsed memories (96 images)
21.92% higher
0.35% lower
Yet occluded-cue retrieval with replay fell from 81.25% to 76.04%. A memory can remain readable while becoming harder to find through a changing encoder. Continuing learning therefore needs to preserve both the content decoded from stored vectors and the route back to them. Replay also worsened reconstruction on the separate larger-frame video diagnostic.
Execution: completedScientific verdict: Coarse geometric reuse and continuation met the provisional criteria; image detail remains limited
Infer a relation from a visual demonstration, reuse it on different content, then compose another relation with the saved VSA result.
Findings and limitations
Qualification correction: accurate relation estimates and the recorded numerical pass did not establish correct rotation in the decoded scene. Structures can be missing, fragmented or incorrectly oriented, not merely blurred. Rendered rotation remains unqualified. The original verdict, measurements and downloadable records are retained as historical evidence.
Fifteen of sixteen demonstrated changes were accepted. Median geometric displacement error, including the refusal, was 0.93% of the image short side. The particular rotation, scale and displacement were inferred from image pairs; the possible geometric family was supplied.
A second operation used only the saved first-stage VSA and a second inferred relation. After two operations, RGB mean squared error was 0.018995 (pixel values scaled from 0 to 1; lower is better) versus 0.057015 for no geometric change through the same number of passes, and 0.048183 when the first change was skipped. Those are reductions of 66.7% and 60.6%; all 32 queries improved against the unchanged comparison.
All five predefined numerical criteria passed, and independent checks reproduced all 416 image comparisons. The first-step comparison against wrong demonstrations remains statistically uncertain.
Supplying the true geometry still produced mean RGB error of 0.011179, compared with 0.000570 for directly encoding and decoding the desired photograph. A subsequent component diagnostic found local rotation failure even with original encoder features and exact geometry. Recovery from source VSA is therefore not required for the failure; this does not isolate every contribution from feature transformation, composition and decoding.
Rotation, uniform scaling and translation are a supplied family, and the two-step procedure is supplied. The system fits numeric relations without neural retraining or acquiring a new operation family. Refused fits use explicit identity fallbacks; the refused example remains in all totals. Desired query images and known geometry are observer-only references in this experiment.
The five historical numerical criteria passed and the first-stage wrong-demonstration confidence interval includes zero. Neither fact establishes intact geometry in the rendered result. Relation fitting and continuation through saved VSA remain separate evidence; the application path is not a qualified visual rotation mechanism. No semantic editing, invented detail, general imagination or runtime promotion is established. The continuing life is unchanged.
Execution: completedScientific verdict: fixed training development diagnostic complete
Distinguish improvement in the learned feature objective from useful decoded predictions, on training examples and separate development videos.
Findings and limitations
Original-feature transport lowers training image error 6.09% but raises development error 5.14%. Recovered synthesis lowers training feature error 32.06%, mostly in the coarse overview, while image error improves about 1% on both training and development. The audit passed all 960 comparisons. Fixed-panel visual review finds recognizable scenes and persistent blur or overlapping moving contours, not consistently coherent middle frames.
Training examples were previously used for learning. Development videos are separate from constructor training but are existing diagnostic data, not a fresh test set. Original encoder features are privileged inputs. Feature error is not image quality or coherence; low error on dark or static regions is not temporal understanding. All outputs pass through VSA before the frozen renderer.
Execution: completedScientific verdict: original feature learning effect below criterion
Test whether the limited temporal-construction result also occurs when endpoint features are supplied directly from the frozen encoder.
Findings and limitations
Original encoder features improve the untrained blend by 2.91% on whole-image error and 4.26% on changing pixels. Against their own original-feature blend, transport changes changing-pixel error by −0.60% and synthesis by +0.19%, with different effects between the two videos. Neither reaches the required improvement on each source. Complete visual review finds no consistent learned repair of doubled moving contours.
The same previously exposed test cases from two videos. Original encoder features are privileged inputs at fitting and evaluation; they do not qualify a VSA-only input architecture. All candidates still recompose VSA before the frozen renderer. This is a bounded diagnostic, not independent validation or a complete causal isolation of the learning limitation.
Cliste is intended to become Sophia. Her earlier graph-based architecture and interfaces must be free to change around what this research establishes.
Perception, partial recall, temporal continuation, and new experience now work together in a bounded persistent loop. The next work returns to acquired cleanup, vocabulary growth and refinement, and transfer across changed views. The bounded cache comparison reduced retrieval cost while preserving the tested behavior. Beyond those lie acquired relations and transformations, learning from consequences, and active perception across senses.
The larger goal is for experience to change what she expects, attends to, and chooses to investigate. Reflection should revise the implications of earlier events and carry useful methods into later activity. Those choices must eventually be learned through consequences.
Fast responses and slower deferred work will have different demands. Their evaluation needs to distinguish mistakes that defeat a use from acceptable approximations, and test when returning to a problem with more evidence helps. Those results should guide the next capability changes and optimization rounds.
Useful cleanup, learned attention, transfer across changed viewpoints and native visual action are still unresolved.
