golemfactory / clay Goto Github PK
View Code? Open in Web Editor NEWGolem is creating a global market for computing power.
Home Page: https://golem.network
License: GNU General Public License v3.0
Golem is creating a global market for computing power.
Home Page: https://golem.network
License: GNU General Public License v3.0
setup.py has only "golem" package specified manually. It will not install subpackages like "golem.network", etc.
You can use setuptools.find_packages()
http://setuptools.readthedocs.org/en/latest/setuptools.html#using-find-packages
Assumptions:
Right now tasks output during computation is written to the console. It should be redirected and written to log files. Those log files may also be send back to the requester and be part of verification process.
If this information is accessible user should be able to see from gui, why specific subtask failed. Was there problem with verification, timeout or user send error message? Right now we have to look for this information on golem console, log or by checking received files.
Describe what should exactly happen when we close Golem node and what should happend when we kill it. Should we write some data to database, kill computation on Virtual Machine, write some states, inform other nodes? Check which elements are implemented and add missing ones.
Both Luxrender and Cycle allow GPU rendering:
Chosen virtualization platform should support OpenCL, is it possible for Virtual Box (or Virtual Box Guest additions)?
Create docker image containing GNR compute node.
Docker image containing Blender might be a good starting point: https://hub.docker.com/r/ikester/blender/
Add support for storing node/application data in user defined folder. E.g. --datadir=/tmp/test-node1
.
By default it should be standard OS-specific "add data" folder. It can be queried using appdirs package.
Implement redundant verification from: https://github.com/golemfactory/golem-rd/wiki/Verification
First approach (abandoned branch) was here: https://github.com/golemfactory/golem/tree/redundant_computation
One potential approach is described below, but any other rational algorithm will be fine.
Algorithm proposal:
We define replication level x and consensus level c > x/2 and max tries m. Tasks send up to x copies of subtasks to different nodes and wait for the results. If c same answers is received the subtask is verified and all answers that are consistent with the consensus are accepted. If consensus is not reached within x subtasks, than additional m - x tasks may be send. If consensus is not reached subtask is marked as failed.
If consensus is reached before all x subtasks are send, then subtask is accepted.
Necessary changes:
Describe what exactly should happen after timeout for task / subtask has been reached. Description should include both provider and requester scenarios.
Golem version of something like: https://stats.ethdev.com/
There used to be some manager applications but they are very old and probably aren't working any more with newer code.
Describe which functionalities should be available to users and which one are only for developers and testers.
Databases, internal lists, etc.
It's quite obvious that user should be allowed to cancel computation, ie. paid for previously computed subtasks, but don't accept any other providers issue.
What is not clear is that if there should be special broadcast method indicating that requester doesn't want to compute this task anymore and information about it shouldn't been kept and broadcast further. It may be useful for very long-standing task but generates some additional complexity.
Additional note: we should be sure that when requester decides to cancel task it pays for all remaining computation, ie. results for the subtasks that has already started being computed. Additional question: what should happened when we added redundant computation and some subtasks has been computed by some nodes but not enough to verify them.
How to propage tasks in Golem network.
Integration with Ethereum protocols seems to be troublesome for now, but we may return to this topic later when p2p protocols will be improved.
Allow user to send .flm files to Golem network and make other Golems to resume computation rather than start them from scratch.
The version of a code from a nanopayment whitepaper should be put into the repository and subaccounts contract should be written. All contracts must be tested in a Golem Private Network as soon as it will start to work again.
After that updated support for new contracts should be implemented in Golem.
Current code is in a Python 2.7.
Extend and improve GNR interfaces:
python gnr-node.py
. It should not import any Qt packages.admmain.py
, grandmmain.py
, main.py
.on wiki
On wiki
Current performance test task was chosen for PBRT renderer (minilight). Maybe we should be choose different task, somehow connected with Blender?
Additionally test task should be connected with environment, not with global config?
https://github.com/imapp-pl/golem/wiki/Verification
There are two issues for advance singular verification:
Add support for UPnP to Golem.
Adding this option will eliminate the need to forward ports by hand for some users. Port forwarding is very uncomfortable process or sometimes not even possible for some users. Having it partially automated will significantly improve UX.
User should be able to choose if she/he wants this option or not. There should be additional flag in config file that would turn on/off this functionality.
We should check if there are any python libraries that support that. The most popular one in C (and used in all p2p application) is miniupnp.
Add new Golem task type with no external dependencies, to be used for testing. Should be customizable to allow different size of input data, output data and level of computational difficulty.
How to inform users about subtask progress? How to measure progress?
For Blender i Luxrender task we should probably monitor logs and find specific strings in logs.
on wiki
When user is testing task and it takes a lot of time, he may close test window and try to test another task or test task again. He's not able to this, because test task directory is locked with another thread. Any solution for this problem will be appreciated.
Implement resource transfer over IPFS, as a replacement for the current methods or as an additional method.
There should be possibility to run Kademlia's find node
function to find a node address using only his id, so adding address may be redundant. On the other hand adding address may be quicker and reduce network traffic.
Read blog post and list of resources from Jeff Coleman. Write a short description and evaluate how useful those methods may be in Golem.
Interface that should allow user to see past and future payments.
What other methods should this interface make available? Should it allow to make payments before expected time? Should it allow to make see Ethereum account balance? Should it allow to send additional money to Eth account?
Right now there are no implemented methods for advance verificaiton on LuxRender. We canno use "part of a subtask" method from other task types. We probably should check luxcompare software that should does exactly what we require (PSNR check of flm files).
How can we test network scenarios (at least simulate many nodes with different IP's)?
We have tried and made some test with imunes
, but this topic is still open.
Previously clients were identified by uuid called client_id. Then we have added identification by public key (called key_id). Task manager and tasks instances still uses client_id (for example to identify if a client has counted a subtask in that task already). This should be replace with a new method.
Rendering tasks create sometimes temporary files, for example Blender task is creating "crop file" that describes a rendering window for a subtask. However there is still a problem with creating and naming a file that is platform independant - right now creating a temp file on Ubuntu generates a file named "" instead of random name.
For tasks there should be a added subtask_id and delay_time as an argument. The name should be also changed from "accept" to "set down" or something similar. The result of this method is a delay value, not a boolean as a name is suggesting.
We should control and fix timeout that are used in tasks. Mainly task timeouts and subtask timeouts, but also some other timeouts that may occur in code (eg. in task computer or task keeper for removing old tasks).
There are some additional important problems with that, ie differences with timestamps in different timezone.
TaskHeader
should have additional field for signature. Otherwise there is a risk that malicious node may produce false task header directed to other node as a requester.
Task_id is unnecessary because task knows it's own id. It should be replaced with subtask_id or list of subtasks_id which will give possibility to send different resources for different subtasks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.