arlodotexe / brain-dump Goto Github PK
View Code? Open in Web Editor NEWIf you had unlimited time and resources, what would you build?
If you had unlimited time and resources, what would you build?
The original thread spawned some interesting ideas, and I'd like to clearly separate them so it can be refined and discussed as needed.
If I had unlimited time and resources, what would I build?
See also:
This one is a bit more "out there" and should have probably been binned, but let's write it down anyway.
.git
folderIdea: Create a general-purpose component that uses a generative AI model to dynamically generate or redesign XAML for templated controls, given a user-supplied text prompt. This could be used in any app that uses the MVVM architecture and could potentially be contributed to a larger community library.
Update: After further consideration, the plan is to focus on creating a custom Templated Control called GenerativeAiControl
. This control will leverage the built-in Template property of the Control base class, allowing the AI to dynamically control the visual structure of the control.
How to Create a GenerativeAiControl
:
GenerativeAiControl
is an abstract class that uses AI to create and modify templates for templated controls.Prompt
property on your control. Keep it simple!Future Considerations:
GenerativeAiControl
is implemented and working well, it can serve as a stepping stone for more advanced features, such as allowing the AI to manipulate properties or call methods directly. This could be done by extending the GenerativeAiControl
or by integrating it with other components or systems.Challenges:
Warning
This is a gpt-assisted draft for a potential idea. It is not an announcement and does not reflect official plans.
We've built a vibrant community on Discord, bringing together project managers, developers, designers, translators, beta testers, enthusiasts, press, Microsoft employees, MVPs, and everyday users. We're proud of what we've achieved so far, but we're ready to think bigger.
Our vision is to create a platform where any creator or publisher can invite and interact with their userbase. By successfully joining a community, a publisher's projects become visible to everyone in the community, and a user is able to see all participating projects. Communities set chat moderation requirements for projects that join.
This platform will break free from the constraints of Discord and create an independent social graph of apps and projects. That can be safely accessed and extended by anyone, and isn't subject to link rot.
The UX of the app is still undecided. The project profiles may take queues from Facebook or LinkedIn. Publisher and Project chat rooms may resemble (discord-like) well known chat applications, and Mastodon-style moderation and identity verification.
The new platform will be fully open source and powered by peer-to-peer technologies, reducing costs and increasing resilience. Here's how it will work:
The platform will also include an SDK for project owners to interact with their community directly in their project, including embedding chats, analytics, and other community-focused features directly into their apps. We'll also use the SDK's plugin system build integrations like GitHub, Discord, and Partner Center.
This new platform will provide numerous benefits:
This is a significant project, and we're planning to roll it out in several phases over the next year or so:
We're excited about this next evolution of our community and look forward to sharing more updates as we progress. Stay tuned!
🚨 This has been ported from Arlodotexe/strix-music#201
Strix has pledged to make an effort to keep the project "perpetually preserved". Put another way, we should avoid relying on things with a single point of failure. The repo should contain everything needed to operate the project, and a release should contain a repo backup + all dependencies expected to be available to build the project, published to IPFS.
In order to keep things resilient, we use IPFS extensively to back up and host the content on our website (docs, WASM app, app binaries, build dependencies, source backup, published nuget packages, etc)
As a result of all this, Strix has a lot of build scripts. They contain everything needed to create a release, and there's a guide[1], [2] to help string them together into a working solution that can download build dependencies, generate documentation, generate our website, create release tags, generate changelogs, and more (with a few more coming in #203 and #202).
For something that should be the equivalent of "yes this commit built successfully", DevOps and GitHub Actions may be extremely convenient, but it's complex, expensive and doesn't play into the spirit of Git.
Further, git itself has no built-in features that enable contributors to verify "yes, this commit built successfully".
We can do better.
(proposal has been amended, see comments)
(Prerequisite: #203)
Leveraging our existing build scripts and Git's already decentralized nature, we can combine Git with IPFS to create a distributed build validation system.
Since PowerShell is used in a lot of CI scenarios (and we've already leaned into it because of this), we'll be developing a PowerShell Module designed to work with Kubo.
The symbiotic relationship between humans and technology has always been about transcending limitations. From the wheel to the printing press, to the internet, and now AI – each innovation has expanded human capabilities. Using AI to extend one's cognitive reach is just the latest chapter in this story.
The dream is to achieve a harmonious balance where technology feels like a natural extension of oneself, enhancing capabilities without overshadowing the human essence.
As technology evolves, and as we understand more about the human psyche and neurology, the pathways to achieve this balance will become clearer. The journey towards this future is filled with challenges, but also with immense possibilities.
Here, we imagine the potential of a symbiotic relationship between human and artificial intelligence, what it may look like, and how it can be achieved.
Note
This document will be updated as we refine it from a dream into a plan. We are currently working on an Exocortex prototype here that you can inspect and try.
Definition: The Exocortex is a remembrance agent, a generative agent specialized in remembering a narrative of observed events.
Vision: A timestamped and vectorized timeline of experiences and notes. Key to this vision is the concept of the memory stream, a dynamic entity that evolves with new experiences. It's designed to provide a sense of continuity, using time-ranked retrieval of related memories to provide context for new experiences.
Enhancing communication: The immediate application and likely testing grounds will be to create a timeline of real events and notes in a specific domain to create a basic prototype, with the aim to identify and eliminate communication bottlenecks. Through this, constructs can collaborate or exchange knowledge and experiences autonomously.
Privacy: First priority, never to be undermined. All computation should be performed on-device, and all communication should be done using peer-to-peer technologies like IPFS to keep data ownership in the hands of users. The exact AI model used should be swappable by the end user. Users should be able to create separate exocortexes for work and personal life, with the option to combine them together.
Initial Memory Values:
Final Retrieval Score: The retrieval function scores all memories as a weighted combination of recency, importance, and relevance. Scores are normalized to the range of [0, 1] using min-max scaling. In the current implementation, all weights (𝛼) are set to 1. The top-ranked memories that fit within the language model’s context window are included in the prompt.
Continuity Across Memories: As new events or memory entries occur, the system retrieves past memories based on factors like time, relevance, and importance. This provides context for the new memory entry. Objective observations become personalized to past experiences, especially the immediate past.
Meta-Observations: Removes the need to include the full memory transcript in the context window.
Time-synchronized first-person observations of events are ideal for incorporating into the memory stream.
Audio-based Memories: Utilizing the WhisperX library for audio transcription. Includes Speaker Diarization, does not include speaker detection. We'll need to know when the user is the speaker, at minimum.
Image-based Memories: Employ the LLaVa library for image description. Multimodal LLMs are capable of rich text descriptions, though not always with the greatest accuracy. We're prototyping, so we'll roll with it while the field keeps moving. Human visual memory isn't perfect either.
Future: Transition to hardware solutions, like smart glasses with camera and mic or a body cam.
Additionally, as noted in the paper "Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction"1, structured scene descriptions allow models to predict what might happen in-between or in subsequent frames. This has significant potential in various applications that can be explored later.
A Self-Organizing Map (SOM) is a type of artificial neural network that is trained using unsupervised learning.
Mimicking Human Sleep Cycles: During 'SOM Sleep', the Exocortex would use SOMs to reorganize and consolidate memories, akin to how human brains are believed to consolidate memories during sleep. This process could involve strengthening important connections between memories and weakening or pruning less important ones.
Contextual Integration: During SOM Sleep, the Exocortex could use SOMs to integrate recent memories with old ones, updating and recontextualizing short-term memories into long-term memories based on new information and reflections.
Memory Pruning and Cleanup: SOM Sleep could also involve cleaning up the memory storage, identifying and safely removing redundant or irrelevant memories, similar to how our brains are believed to prune unnecessary connections during sleep.
Definition: O.M.N.I.A. (Operating Matrix of Networked Intelligent Avatars) is a digital environment constructed with natural language observations of the local environment. In the original paper "Generative Agents: Interactive Simulacra of Human Behavior."2,Smallville was crafted by hand. By extracting environmental or synthesizing data for one or more exocortex, we can roughly simulate how a construct might interact with it.
This concept opens the door for constructs to act within a real or simulated space, and respond to it in meaningful ways, even engaging in proactive assistance personalized to the user's needs.
Normally when interacting in this world, entities other than yourself are driven by generative agents. Substituting those agents with the construct of real people (inhabiting descriptions of their own local space) will enable intelligent exchange of information in a highly dynamic and meaningful way, without violating the rights and privacy of the parties involved.
The foundations are emerging technologies, and it will require extensive research and advancements in the field of machine learning to pull off. Luckily, all the pieces are there and ready to pick up.
Vaishnavi Himakunthala and Andy Ouyang and Daniel Rose and Ryan He and Alex Mei and Yujie Lu and Chinmay Sonar and Michael Saxon and William Yang Wang (2023). Let’s Think Frame by Frame: Evaluating Video Chain of Thought with
Video Infilling and Prediction. arXiv preprint arXiv:2305.13903. ↩
Joon Sung Park and Joseph C. O'Brien and Carrie J. Cai and Meredith Ringel Morris and Percy Liang and Michael S. Bernstein (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv preprint arXiv:2304.03442. ↩
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.