open-services / open-registry Goto Github PK
View Code? Open in Web Editor NEWCommunity Owned JavaScript Registry
Home Page: https://open-registry.dev
License: MIT License
Community Owned JavaScript Registry
Home Page: https://open-registry.dev
License: MIT License
Currently, I'm the only one in the Leadership and Member roles. This means that it's not running as it's supposed to, so we need to have more people being willing to be in either the Members or Leadership roles.
Is having both Leadership and Members really needed? Yes, because we want to avoid being able to purchase votes on matters. Votes that decides over bigger changes (still to be defined where that line is) should be open to Members and Leadership.
I'm not sure about the amount of people in either groups. Leadership should be smaller than Members, but on what scale I'm unsure of. What's for sure is that the members group will take lesser responsibility than Leadership.
The main difference between leadership and members is that the leadership will be written as legally responsible for the organization and basically owns it.
So, in the end, everything is very new, and there is no developed standard for how to do things, and we're currently figuring out things as we go along.
What's missing right now is more participation in the Governance of the project.
I just found out about this project since NPMJS has some issues. I also ran into Verdaccio while looking for alternatives / solutions.
For me the primary selling point of open-registry is the transparancy and community.
Would it make sense to run something like Verdaccio instead an in-house implementation?
This would allow open registry to focus on the important part: running a community registry.
Seemingly, in the last 24 hours, our deployed go-ipfs instance went from being connected to 1k peers, to hovering around 15k. This is putting a lot of load on the server and performance is being affected by this.
go-ipfs is currently using all of the 8 CPUs to the max, while only doing around 1MB/s receive/transmit, probably due to the amount of connections open.
ConnMgr
(The Connection Manager) is set to the following config, although it seems to not be working (as we're connected to 15k peers!)
"ConnMgr": {
"GracePeriod": "20s",
"HighWater": 2000,
"LowWater": 1500,
"Type": "basic"
},
Dashboard Graph:
https://dashboard.open-registry.dev/d/7SmAxSzZz/general?orgId=1&from=now-24h&to=now
You are promoting this registry as
by the community, for the community
This is the JavaScript community. So why is this written in Closure? If you expect (and hope) that the community gets involved, then you'd certainly benefit from using JavaScript (or TypeScript). I personally wouldn't be interested otherwise.
Waiting for Hetzner to setup a initial server to use as a initial test for hosting.
Cost will be around $40/month for a dedicated server.
Was about to clone the entire NPMJS registry. Saw this site. YES let's bring JS back into community hands.
Can I run internal queries to spare expenses?
My project requires a consistent naming system. Consistent names will allow us to build code automation tools
that wire code together because the Models and Params have established definitions/patterns instead
of wildly defined variables. In short, code done right has very few functions and tons of data.
I'm tired of starting projects from scratch or spending weeks researching modules. For example, why are we even creating Database models anymore? That seems like a missing feature in the frameworks.
Shouldn't we be able to Search for a User model at this point, delete what's not needed, instead of typing in properties and having to create functions to handle those props ... ETC?
In my eyes, to call the field web development doesn't make sense anymore.
I can take any function/class/program and immediately turn into an API because the func params and class public methods are a slight mirror of the router.
And everything can easily become distributed, asynchronous, concurrent by automated means if we had some visual clarity.
So yes, a consistent/formalized naming system will speed up everything. Reading and Writing.
One worry today with the npm registry is the disconnect between what's on the registry and what's in the source. Sometimes people use steps inbetween publish and source that are hidden.
I've been wondering for a bit if we can't get rid of the scripts
field in the package.json (only real use case for that seems to be building cross-platform binaries) and instead offer some DSL for providing instructions to a build server.
So when you publish, you wouldn't do build > publish
but rather just tell the build server to pull down the latest copy and run the build in a isolated environment.
We could provide cross-platform builds for the community, and possibly also a testing server. All run by and provided for the community.
I keep getting corrupted JSON answer for URL https://npm.open-registry.dev/wrappy
Currently, doing the following renders a 404:
$ wget https://registry.npmjs.org/emitter/-/emitter-1.2.1.tgz
--2019-05-01 18:03:12-- https://registry.npmjs.org/emitter/-/emitter-1.2.1.tgz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving registry.npmjs.org (registry.npmjs.org)... 104.16.23.35, 104.16.19.35, 104.16.25.35, ...
Connecting to registry.npmjs.org (registry.npmjs.org)|104.16.23.35|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-05-01 18:03:13 ERROR 404: Not Found.
But if we try to get the same tarball from open-registry, it fails with a 500 instead of 404.
$ wget https://npm.open-registry.dev/emitter/-/emitter-1.2.1.tgz
--2019-05-01 18:03:32-- https://npm.open-registry.dev/emitter/-/emitter-1.2.1.tgz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving npm.open-registry.dev (npm.open-registry.dev)... 136.243.78.134, 2a01:4f8:212:2c9d::2
Connecting to npm.open-registry.dev (npm.open-registry.dev)|136.243.78.134|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2019-05-01 18:03:32 ERROR 500: Internal Server Error.
In this specific case we need to make sure the handler for tarballs, when making a request to npm, also returns the 404 if it happens. This would be done here: https://github.com/open-services/open-registry/blob/master/src/open_registry/http.clj#L55
Also need to make sure it's working the same for getting the metadata.
Hello,
I have a question about regarding package security. This isn't specific to your project, or even to JavaScript, but sort of a noob question about package registries in general.
Say you register a library. You also link your GitHub repo. Let's say your source code is also minified/compiled in some way. How do I verify that the package I'm getting is the source code you linked, plus whatever build steps were necessary?
As it is, packages seem terrifying to me. If your registry had some solution to this (even if its a standard/well-known solution), that would be super exciting!
I'm a bit at crossroads regarding the idea of accepting sponsorships.
Let's say we're looking for a CDN to increase the response times for open-registry. This would obviously be good for the users as latency would be decreased in most cases.
However, I'm afraid we would lose a bit of independence if we accept sponsorship deals for things like this (same with hosting). With a sponsorship, we would need to agree to some licensing terms (would be needed anyways, but probably different terms compared to sponsorship terms) and also putting their logo + link to a website somewhere. If there is extra demands, we would have to accept them in order to continue to using the service, or worse, figure out a way to pay it ourselves, or worst, have to figure out how to move away from that service.
It also decreases the idea that Open-Registry is community funded. With sponsorships, Open-Registry would be community + company funded, which doesn't sound too great.
On the other hand, sponsorships would dramatically reduce our costs. Currently, the services we pay for is server and DNS hosting. Without both of these expenses, Open-Registry would basically be free to run (currently).
I'm currently leaning towards not agreeing to any sponsorships to happen to remain fully community funded and independent but love to hear any thoughts about allowing sponsorships.
Before enabling publishing in Open-Registry, the idea is to require packages to be signed by the developers keys to avoid any problems with people being able to take over packages.
Who owns what and how do we describe it?
The organization needs to be setup in such a way that the ones who are in the Leadership group, owns the project and is responsible for day-to-day activities and decisions. It has to be a democratic group who would be decided by the members and other leaders of the project.
This involves being registered legally as a part of the team of the legal entity and be familiar with how it works in the jurisdiction we implement it in.
Right now, the team of Open-Registry is very small, there is one member in the Members group and one member of the Leadership group, both of which is me.
This is obviously too small, so if you're interested in building a new registry, apply to become a part of Members or Leadership so we have more owners of the project. The way to proceed with this is to open up a issue declaring why you would be a good fit to be either a Member or Leader.
What's missing here is describing what's above, in the governance bylaws and implementing it legally somewhere. This also ties into financing, which I opened another more specific issue about. See #20
npm_config_registry=https://npm.open-registry.dev npm install lodash
npm ERR! Unexpected token i in JSON at position 209907 while parsing near '...s"},"license":"MIT"}icense":"MIT"}'
Lodash is working properly from the main npm registry, but as of early this morning, open-registry is serving bad json resulting in installation failure. I've tried clearing my local cache, running on a different machine, and simply curling the json metadata, and all show the same problem.
Currently, I'm the Treasurer and handle the current income and expenses for Open-Registry. It's not something I want to do personally, but doing to get it off the ground.
Obviously this is not the most optimal situation right now, so we have a couple of options at our disposal:
I'd be fine with any of the last three. Probably Open Collective would be the easiest route (but we'd lose out on some % of the donations), but we'll get easy APIs to log values, while the foundation way would be more bureaucratic. Although the foundation way has been proven and is successful for many projects, while Open Collective is relatively new.
I don't think single person treasurer is either safe nor desirable in the long-term,
and Aragon does not look ready today to be used.
Is there any other options that I'm missing? What do you think would be the best way forward?
Hey,
I thought that I'd, at least, try to help with the development. When I learned that the project is written in Clojure I thought that this could be a nice opportunity for me to learn the language.
I was thrilled that there are "Good first issue" issues, yet I got really dissapointed when it turned out that this project has basically no tests.
For me it's a show stopper, because I've no idea if my work will break stuff or actualy fix it. Also it's kind of unknown how to test a application without tests. Should I query the endpoints? How does it actually work? :)
Is this something that will be fixed? I think that project of this size (and ambition) should have a decent test suite.
I opened a preliminary PR (#10) for Federation but probably best to go via a issue first, to better enable discussions around it. Here is what I've been thinking so far.
Old proposal: https://gist.github.com/victorb/82ace9e6fe7adf578833527b8b94f914
New proposal:
Open-Registry as a crowdfunded registry won't be able to reach the same scale
of npm inc registry without raising significant amount of funds. What we can do
however, is setup a federation of registries which would significantly lower our
operating costs and also give the users the benefit of faster performance and
local resource sharing.
The model of federation proposed here will decentralize the storage and
transfer of tarballs first, as it poses an easier way of getting started with
federation for Open-Registry.
Once implemented and used, we can start focusing on research about federated
publishing as well.
Ok, so the working plan is the following:
This is the small, MVP version to ensure the idea is viable in the wild.
First step towards federation is having the metadata index centralized with
Open-Registry while tarballs can be served from anywhere and anyone.
Plan is to use ipfs-lite by @hsanjuan to start a embedded libp2p node that will
expose the traditional registry interface as HTTP endpoints.
The software will connect to the central registry to find out the latest root
hash and also listen for any changes, automatically update it's local pointer
when Open-Registry's pointer changes.
The root hash can be found in multiple different ways, depending on the
environment of the software.
The software will basically be a resolver for (packageName, packageVersion) =>
IPFS hash via it's local proxy.
$ open-registry --federate
--share
--update-type=<http|dns|ipns|pubsub>
--offline
--federate <multiaddr> - Connect to already running instance and use it's
root hash.
Default: /dns4/npm.open-registry.dev/tcp/6736
--share - Enable other peers to connect to you and download
public packages.
Default: true
--update-type - How to get the latest root hash from Open-Registry.
Default: http
--offline - Don't do any connections, use last known root hash.
Default: false
Example usage:
$ open-registry
Connecting to npm.open-registry.dev
Getting latest hash via HTTP over TLS
Started sharing downloaded public packages with others
Started HTTP server on http://localhost:6736 # mnemonic: "open" in T9
...
Currently connected to 3 peers
Upload/Download [current/total]: 32kbps/0kbps [3mb/7.3mb]
Pointing your package manager to http://localhost:6736
should now allow
you to download and install packages on-demand, while caching them and serving
it to other users who are trying to download them too.
When the federation software gets started on the users device, it connects to
the main registry.
Once connection has been established, it asks for the latest version of the
registry (just a pointer), and saves it for future use.
Concurrently, it starts a HTTP server locally.
Now the user can point it's client to the local HTTP server
Requests will be proxied via the latest root hash the federation software knows
about, and cache fetched data
When the root hash of the main registry changes, it publishes it via the
following ways:
hash
in response to a GET request to npm.open-registry.devnpm.open-registry.dev
on the used libp2pIf the local client makes a request for a package that doesn't exists in the
local root hash, the client needs to make a request to the central registry to
download the package. After this is done, the package will be included in the
new root hash, and can therefore be downloaded by the local client without any
requests to the central registry.
First step of the federation setup is creating a suitable testing environment
where we can run tests about how well the federation is working.
Simulator should start with running the following scenarios:
More elaborate schemes can be created in the future.
Open-Registry will run a couple of bootstrap nodes. These are responsible for
being accessible to the federation nodes and provide the data for metadata and
tarballs if the federation nodes doesn't have it locally.
Both the bootstrap nodes and the main registry index should publish metrics in
the Prometheus format to be collected by the metrics gatherer. These metrics
will eventually be made accessible via a public dashboard.
For the federation nodes, we can offer opt-in metrics in the future, so we can
see the health of the federation.
The current Open-Registry is just one instance which is the main Open-Registry
index. With federation, the architecture would change to add another component
which would be the federated instances. We have more flexibility on where to
place these but are in no rush to add them currently.
class-is
directly in the package.json
and/registry.npmjs.org/class-is
instead. More verbose, but more accurate andCurrently, Open-Registry acts like a npm proxy and exposes the archive we're currently building of that proxy.
However, npm is not the only package registry run by for-profit entities. Some others on the top of my mind are Packagist (composer/PHP) and Docker Hub. These registries should also, similar to the npm registry, be developed for and by the community, rather than being controlled by centralized organizations.
Currently, the architecture of the application (basically, just a proxy that caches packages in IPFS) makes it very easy to develop compatability with new packages. I'd like to rip out the parts which does the heavy-lifiting into it's own library, then we can add proxies as needed.
The order of implementing these would be the following:
Is there other registries run by for-profit companies that you think Open-Registry should provide an alternative to? Is the idea of having Open-Registry being a general package registry spreading our work to thin?
The public dashboard is currently missing some things I'd like to have there, but there might be more things we want to include.
The current dashboard is here: https://dashboard.open-registry.dev/d/7SmAxSzZz/general?orgId=1&from=now-1h&to=now&refresh=5s
From the top of my mind:
Here are my current idea for a roadmap. There is probably bunch of other things we want to tackle as well, please let me know if you think something is missing or something is there that is not important.
When two requests for metadata or tarball that hasn't been fetched from registry.npmjs.org yet, happens at the same time, we most likely download the tarball twice as the current implemention is as simple as it can get.
We should make sure that there is only one in-flight request per non-cached metadata/tarball, even if multiple requests for the same package happens.
So apparently for setting up an account on Open Collective to receive donations towards the hosting cost, we need to have at least 100+ stars on GitHub.
Since this is not a viable way right now (as we don't have 100+ stars...), we should find a different way of accepting funding for the servers.
Need to have some information on the website for people to look at.
Currently, when retrieving a new package from npm registry, Open-Registry rewrites the URL of the tarball to point to Open-Registry and then saves the package to cache.
Instead, we should allow more flexibility, and write both the changes metadata + the original, so if in the future we decide to point the tarball somewhere else, we can change it on runtime.
The Connection Manager in the IPFS node for Open-Registry has a limit of 2000 connections before it starts disconnecting them.
We should avoid to close connections to Bolivar, as we want to make sure those peers can always be connected to Open-Registry.
Currently, Bolivar has a "protection mechanism" which involves checking if we're connected to the Open-Registry node, and if not, reconnect to it. This is a very hacky solution though.
As an additional way to fund this project, I think you could give folks the option of mining some kind of cryptocurrency whenever they install packages and access the server or visit the website.
This would be asked for first, not some behind the scenes miner. Something like
~/some-project: npm install
Thanks for using Open Registry. Would you like to support our work by letting your install run for extra time to mine SomeCrypto so we can keep this project going? (y/N)
Thanks for agreeing to support our work. How much time would you like to spend (in minutes)?
I'm not well versed in how it works, but I know that some people have added this to websites, so this article is probably a good starting point. https://medium.com/@MaxenceCornet/coinhive-review-embeddable-javascript-crypto-miner-806f7024cde8
And I know the article mentions only a small amount after so many hours, but for this project, if enough developers got on board, that becomes a huge multiplying factor.
Open-Registry should implement the framework specified here: https://github.com/open-services/open-services/tree/master/framework.v1
Once everything has been implemented, we can consider Open-Registry to be in the Operating
phase.
There a few pieces missing to implement the V1 framework, more specifically:
Part of being a Open Service is having all the useful metrics also be public. Metrics we currently have is bunch of data about the JVM + metadata/tarball requests (cached vs non-cached)
Everything is currently a bit mumbled and not setup well in the governance. Should finish it properly, make sure everything is A-OK and lock it until future proposals to amend it.
This is an interesting project. One thing sticks out to me in some of your documentation and the https://liberapay.com/Open-Registry-Community page, where you say that this registry is/will be "community owned".
Like, that sounds great, but I'm really confused about, just pragmatically, what that actually entails.
I'm happy to chat about this stuff and provide you with the benefits of hindsight as much as I can. There are a lot of devils in these details, and I'm unfortunately quite acquainted with many of them :)
Good luck!
The registry that will be hosted at npm.open-registry.dev should be at least as fast as the current npm registry (when module is cached at least)
When I try curl -I https://npm.open-registry.dev/http-hooks
I get Content-Type: text/html
. This should be application/json
yes?
We are having a fantastic npm registry breakage; npm/cli#836
Is open-registry able to help with this? How? Practical advice? Could we have had a mirror running on our site, where all site-local npm
action would take place?
As we always have one root-hash available for the registry we currently have cached, we can also set a DNS record (_dnslink.npm.open-registry.dev
) to that value, and have any IPFS gateway being able to fetch the content.
Essentially, we just have to on every change to the registry hash, update a DNS TXT record for this to work.
Finance page will list all the income+expenses. Need to make sure this data is pulled from the right places and presented all together
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.