The symbiotic relationship between humans and technology has always been about transcending limitations. From the wheel to the printing press, to the internet, and now AI – each innovation has expanded human capabilities. Using AI to extend one's cognitive reach is just the latest chapter in this story.
The dream is to achieve a harmonious balance where technology feels like a natural extension of oneself, enhancing capabilities without overshadowing the human essence.
As technology evolves, and as we understand more about the human psyche and neurology, the pathways to achieve this balance will become clearer. The journey towards this future is filled with challenges, but also with immense possibilities.
Here, we imagine the potential of a symbiotic relationship between human and artificial intelligence, what it may look like, and how it can be achieved.
Note
This document will be updated as we refine it from a dream into a plan. We are currently working on an Exocortex prototype here that you can inspect and try.
The Exocortex: An Echo of Experiences
Definition: The Exocortex is a remembrance agent, a generative agent specialized in remembering a narrative of observed events.
Vision: A timestamped and vectorized timeline of experiences and notes. Key to this vision is the concept of the memory stream, a dynamic entity that evolves with new experiences. It's designed to provide a sense of continuity, using time-ranked retrieval of related memories to provide context for new experiences.
Enhancing communication: The immediate application and likely testing grounds will be to create a timeline of real events and notes in a specific domain to create a basic prototype, with the aim to identify and eliminate communication bottlenecks. Through this, constructs can collaborate or exchange knowledge and experiences autonomously.
Privacy: First priority, never to be undermined. All computation should be performed on-device, and all communication should be done using peer-to-peer technologies like IPFS to keep data ownership in the hands of users. The exact AI model used should be swappable by the end user. Users should be able to create separate exocortexes for work and personal life, with the option to combine them together.
Memory Stream Architecture
Initial Memory Values:
Recency: Each new memory is timestamped upon creation. Recency decays over time, using an exponential decay function (e.g., decay factor of 0.995) based on the number of hours since the memory was last retrieved.
Importance: Assigned an initial score when created, indicating the poignancy of the memory on a scale from 1 (mundane) to 10 (extremely poignant).
Relevance: Initialized based on the context in which the memory was created, and updated based on the similarity between the memory’s embedding vector and the current context’s embedding vector.
Final Retrieval Score: The retrieval function scores all memories as a weighted combination of recency, importance, and relevance. Scores are normalized to the range of [0, 1] using min-max scaling. In the current implementation, all weights (𝛼) are set to 1. The top-ranked memories that fit within the language model’s context window are included in the prompt.
Continuity Across Memories: As new events or memory entries occur, the system retrieves past memories based on factors like time, relevance, and importance. This provides context for the new memory entry. Objective observations become personalized to past experiences, especially the immediate past.
Meta-Observations: Removes the need to include the full memory transcript in the context window.
By treating memory recollection as a new observation, it can add new context to old memories without overwriting them, and it boosts the odds of being recalled again in the near future, mimicking an organic working memory.
Doing this also changes how things are remembered through the lens of the active context.
This is reminiscent of how human memory works: recalling a memory can change how it is remembered, and the act of remembering can itself become a new memory.
Automated Memory Ingestion
Time-synchronized first-person observations of events are ideal for incorporating into the memory stream.
Audio-based Memories: Utilizing the WhisperX library for audio transcription. Includes Speaker Diarization, does not include speaker detection. We'll need to know when the user is the speaker, at minimum.
Image-based Memories: Employ the LLaVa library for image description. Multimodal LLMs are capable of rich text descriptions, though not always with the greatest accuracy. We're prototyping, so we'll roll with it while the field keeps moving. Human visual memory isn't perfect either.
Future: Transition to hardware solutions, like smart glasses with camera and mic or a body cam.
Additionally, as noted in the paper "Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction"1, structured scene descriptions allow models to predict what might happen in-between or in subsequent frames. This has significant potential in various applications that can be explored later.
Memory Consolidation (SOM Sleep)
A Self-Organizing Map (SOM) is a type of artificial neural network that is trained using unsupervised learning.
Mimicking Human Sleep Cycles: During 'SOM Sleep', the Exocortex would use SOMs to reorganize and consolidate memories, akin to how human brains are believed to consolidate memories during sleep. This process could involve strengthening important connections between memories and weakening or pruning less important ones.
Contextual Integration: During SOM Sleep, the Exocortex could use SOMs to integrate recent memories with old ones, updating and recontextualizing short-term memories into long-term memories based on new information and reflections.
Memory Pruning and Cleanup: SOM Sleep could also involve cleaning up the memory storage, identifying and safely removing redundant or irrelevant memories, similar to how our brains are believed to prune unnecessary connections during sleep.
OMNIA: An Echo of Environments
Definition: O.M.N.I.A. (Operating Matrix of Networked Intelligent Avatars) is a digital environment constructed with natural language observations of the local environment. In the original paper "Generative Agents: Interactive Simulacra of Human Behavior."2,Smallville was crafted by hand. By extracting environmental or synthesizing data for one or more exocortex, we can roughly simulate how a construct might interact with it.
This concept opens the door for constructs to act within a real or simulated space, and respond to it in meaningful ways, even engaging in proactive assistance personalized to the user's needs.
Normally when interacting in this world, entities other than yourself are driven by generative agents. Substituting those agents with the construct of real people (inhabiting descriptions of their own local space) will enable intelligent exchange of information in a highly dynamic and meaningful way, without violating the rights and privacy of the parties involved.
Conclusion
The foundations are emerging technologies, and it will require extensive research and advancements in the field of machine learning to pull off. Luckily, all the pieces are there and ready to pick up.
Footnotes
Vaishnavi Himakunthala and Andy Ouyang and Daniel Rose and Ryan He and Alex Mei and Yujie Lu and Chinmay Sonar and Michael Saxon and William Yang Wang (2023). Let’s Think Frame by Frame: Evaluating Video Chain of Thought with
Video Infilling and Prediction. arXiv preprint arXiv:2305.13903. ↩
Joon Sung Park and Joseph C. O'Brien and Carrie J. Cai and Meredith Ringel Morris and Percy Liang and Michael S. Bernstein (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv preprint arXiv:2304.03442. ↩
Strix has pledged to make an effort to keep the project "perpetually preserved". Put another way, we should avoid relying on things with a single point of failure. The repo should contain everything needed to operate the project, and a release should contain a repo backup + all dependencies expected to be available to build the project, published to IPFS.
In order to keep things resilient, we use IPFS extensively to back up and host the content on our website (docs, WASM app, app binaries, build dependencies, source backup, published nuget packages, etc)
As a result of all this, Strix has a lot of build scripts. They contain everything needed to create a release, and there's a guide[1], [2] to help string them together into a working solution that can download build dependencies, generate documentation, generate our website, create release tags, generate changelogs, and more (with a few more coming in #203 and #202).
Problem
For something that should be the equivalent of "yes this commit built successfully", DevOps and GitHub Actions may be extremely convenient, but it's complex, expensive and doesn't play into the spirit of Git.
Further, git itself has no built-in features that enable contributors to verify "yes, this commit built successfully".
We can do better.
Proposed solution
(proposal has been amended, see comments)
(Prerequisite: #203)
Leveraging our existing build scripts and Git's already decentralized nature, we can combine Git with IPFS to create a distributed build validation system.
Since PowerShell is used in a lot of CI scenarios (and we've already leaned into it because of this), we'll be developing a PowerShell Module designed to work with Kubo.
First draft:
Scripts
Long running script for code owners
Can be any machine controlled by someone with full repo permissions (code owners, volunteers)
Performs tagging autonomously on validator's behalf
May require extra abuse prevention
Coordinates over IPFS with other running instances of this script
Run and done script for build validators
Anybody, any machine (maintainers, contributes, traditional CI)
Assumes limited repo permissions
Wants to validate that they can successfully build a commit
Configuration:
What command to use for build validation (as of the built commit)
Define glob paths of built content to upload to IPFS (as of the built commit)
Run and done script to check commit validation
List the validations on a commit (show data)
Should be useful to check against PRs
Node settings
External node
Custom IPFS API url
Embedded node
Download and run temporary IPFS node
Custom repo path
Custom go-ipfs config file. Allows for fine-grain control over the embedded node.
Other
User-defined script to validate a folder of build assets
This is a gpt-assisted draft for a potential idea. It is not an announcement and does not reflect official plans.
We've built a vibrant community on Discord, bringing together project managers, developers, designers, translators, beta testers, enthusiasts, press, Microsoft employees, MVPs, and everyday users. We're proud of what we've achieved so far, but we're ready to think bigger.
Our vision is to create a platform where any creator or publisher can invite and interact with their userbase. By successfully joining a community, a publisher's projects become visible to everyone in the community, and a user is able to see all participating projects. Communities set chat moderation requirements for projects that join.
This platform will break free from the constraints of Discord and create an independent social graph of apps and projects. That can be safely accessed and extended by anyone, and isn't subject to link rot.
The UX of the app is still undecided. The project profiles may take queues from Facebook or LinkedIn. Publisher and Project chat rooms may resemble (discord-like) well known chat applications, and Mastodon-style moderation and identity verification.
The New Platform
The new platform will be fully open source and powered by peer-to-peer technologies, reducing costs and increasing resilience. Here's how it will work:
Users can join existing "communities" if they meet the join requirements and have a join link. Users can start a "community" to unite a subset of other users. Users will have unique identifiers to differentiate between users with the same username.
Projects are created by publishers and can create "channels" to interact with their users. Private projects will be hosted on private swarms. Public projects can be part of communities and must abide by the moderation requirements of those communities.
Publishers are created by users and can create projects. They can be part of communities and may receive visits from other members of those communities.
Communities are groups of users, projects, and publishers created, managed and moderated by one or more user volunteers. Communities can set their own rules, moderate for project and publisher channels, and block users, projects, publishers, or entire communities if necessary.
The platform will also include an SDK for project owners to interact with their community directly in their project, including embedding chats, analytics, and other community-focused features directly into their apps. We'll also use the SDK's plugin system build integrations like GitHub, Discord, and Partner Center.
The Benefits
This new platform will provide numerous benefits:
Distributed Model: We'll leverage peer-to-peer technologies to reduce costs and increase resilience.
Trust and Safety: We can take cues from Mastodon's approach to handling abuse, enabling fully self-hosted self-governance of communities.
Interactivity: Project owners will have an SDK to interact with their community directly in their project, including embedding chats, analytics, and other community-focused features into their apps.
Communication is key: The platform connects creators to their userbase, connects users to their favorite creators, and promotes organized and organic discovery of new communities and projects for creators and users alike.
The Timeline
This is a significant project, and we're planning to roll it out in several phases over the next year or so:
Phase 1 (1-2 months): IPLD Database Migration and Backend Preparation
Phase 2 (6-8 months): Building the Distributed App
Phase 3 (2-3 months): Initial Testing and Importing accounts from old UWP Community backend
Phase 4 (3-4 months): Development of Additional Features and Final Testing
We're excited about this next evolution of our community and look forward to sharing more updates as we progress. Stay tuned!
Idea: Create a general-purpose component that uses a generative AI model to dynamically generate or redesign XAML for templated controls, given a user-supplied text prompt. This could be used in any app that uses the MVVM architecture and could potentially be contributed to a larger community library.
Update: After further consideration, the plan is to focus on creating a custom Templated Control called GenerativeAiControl. This control will leverage the built-in Template property of the Control base class, allowing the AI to dynamically control the visual structure of the control.
How to Create a GenerativeAiControl:
The GenerativeAiControl is an abstract class that uses AI to create and modify templates for templated controls.
To limit prompt complexity, the dev specifies types that can be bound or declared in XAML via attributes in the backing code.
A source generator extracts documentation and type information, and supplies it to the AI model to help it understand the semantics of the types.
Set the Prompt property on your control. Keep it simple!
If an existing style template is set, it will be remixed using your prompt.
If no style template is set, the AI will generate a new one.
Future Considerations:
Once GenerativeAiControl is implemented and working well, it can serve as a stepping stone for more advanced features, such as allowing the AI to manipulate properties or call methods directly. This could be done by extending the GenerativeAiControl or by integrating it with other components or systems.
Giving an AI the ability to manipulate any arbitrary property can indeed be powerful, but it's most useful when it's part of a larger system or framework. This allows the AI's capabilities to be leveraged in a controlled and meaningful way.
Challenges:
The AI model would need to be capable of understanding the semantics of XAML and know how to write XAML that uses "pure" mvvm (converters, commands, inpc, no code behind).
Developing this feature would likely require a significant amount of effort and could potentially impact the app's performance.
Locally runnable language models are not yet up to the task.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent