Comments (11)
@rohanarun Can you please stop spamming your message under EVERY Issue. Thanks
from self-operating-computer.
tried setting grid_interval as 100 instead of 500.
still no success.
it guess as 50 93 which is again not correct.
from self-operating-computer.
@daaniyaan I tested today another apporach. I hink it would be best to use a multi-step-apporach.
- Use a wider Grid to let GPT Vision define Areas of Intereset and return wider coordinates.
- Use these coordinates to crop the screenshot and let GPT Vision do a better guess of find the needed buttons etc.
With a wide Image as the fullscreen Images it might be necessary to do this apporach not only once?
from self-operating-computer.
@daaniyaan I tested today another apporach. I hink it would be best to use a multi-step-apporach.
- Use a wider Grid to let GPT Vision define Areas of Intereset and return wider coordinates.
- Use these coordinates to crop the screenshot and let GPT Vision do a better guess of find the needed buttons etc.
With a wide Image as the fullscreen Images it might be necessary to do this apporach not only once?
this could be a good approach. how many times did you test it? noticed improvement?
from self-operating-computer.
Tested quite a lot today. I noticed that it can handle wider grid sizes better.
And this after cropping it to the area of interest:
from self-operating-computer.
Dude I am just a random guy who finds the idea of using got vison interesting. Didn't even contribute a single thing to this repo.
Chill down
from self-operating-computer.
wide Image as the fullscreen Images it might be necessary to do this apporach not only once?
@Mekcyed This is indeed an interesting approach. I recommend checking out this PR that is working on something similar: #57
If you want to work on this initiative, I'd love to see a PR. Essentially you could do a "binary-search" on the screen, cutting it into ever smaller pieces to find the right coordinance to click!
from self-operating-computer.
Sorry about @rohanarun, I've blocked him from this repo now
from self-operating-computer.
@Mekcyed @daaniyaan we created a discord channel for Self-operating computer on our Discord here if you want to join and discuss more. Great seeing all these ideas!
from self-operating-computer.
Let me know if you guys are done with this discuss here and want to move to discord and I'll close this ticket
from self-operating-computer.
I've not head more on this discussion so I'll close this ticket for now. Feel free to open another issue if you have one or join our discord to continue the discussion.
from self-operating-computer.
Related Issues (20)
- [FEATURE] No update instructions?
- [BUG] WINDOWS install not finding gpt-4-with-ocr HOT 5
- [BUG] Unable to activate the virtual environment
- [BUG] Not running on Ubuntu 22.04.4 LTS HOT 3
- CogVLM Support - A better LLaVa
- [BUG] -m gemini-pro-vision asking for OPENAI_API_KEY
- [FEATURE] Add Remote Ollama Capability
- [BUG] Cannot seem to select the right emails to delete.
- [FEATURE] Learning Process HOT 1
- [FEATURE] GUI Interface and further connectivity
- [BUG] operate -m llava return error local variable 'content' referenced before assignment
- [BUG] ModuleNotFoundError: No module named 'pkg_resources' HOT 4
- [BUG] Need GPT-4 ?
- [FEATURE] Azure open AI support HOT 2
- OpenSource free Vision model use Instead of openAI HOT 5
- Github
- [Linux]: X get_image failed: error 8 (73, 0, 1316) [Error] --> cannot access local variable 'content' where it is not associated with a value HOT 2
- [For be deleted]
- [BUG] No such file or directory Xauthority
- [BUG] Brief Description of the Issue
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from self-operating-computer.