cnpm / cnpmcore Goto Github PK

View Code? Open in Web Editor NEW

547.0 26.0 73.0 13.91 MB

Private NPM Registry for Enterprise

Home Page: https://registry.npmmirror.com

License: MIT License

TypeScript 98.42% HTML 1.49% Shell 0.07% Dockerfile 0.02%

npm registry npm-registry nodejs javascript typescript

cnpmcore's Introduction

cnpm

cnpm: npm client for npmmirror.com

Requirements

	Minimum	Recommended
Node.js	14.18.0	LTS

Install

npm install cnpm -g

If you're in China, maybe you should install it from our China mirror:

npm install cnpm -g --registry=https://registry.npmmirror.com

Usage

Support all commands just like npm.

Sync packages from `npm`

cnpm sync [moduleName]

Open package document or git web url

cnpm doc [name]
cnpm doc -g [name] # open git web url directly

Build your own private registry npm cli

npm install cnpm -g

# then alias it
alias mynpm='cnpm --registry=https://registry.npm.example.com \
  --registryweb=https://npm.example.com \
  --userconfig=$HOME/.mynpmrc'

Install with original npm cli

cnpm using npminstall by default. If you don't like symlink mode for node_modules, you can change the installer to original npm. But you will lose the fastest install speed.

cnpm i --by=npm react-native

License

MIT

Contributors

_fengmk2	_dead-horse	_{semantic-release-bot}	_alsotang	_0xflotus	_weakish
_gemwuu	_ibigbug	_qiu8310	_elrrrrrrr	_fossabot	_popomore
_xieren58	_xiaohuoni

This project follows the git-contributor spec, auto updated at Wed Dec 06 2023 21:42:25 GMT+0800.

cnpmcore's People

Contributors

Stargazers

Watchers

Forkers

zian502 xiekw2010 cevio chenpx976 franciskong beace wcl1237 fossabot amorites hartl3y94 wukd blackhole1 junboxiao changsongyang yang200258 leo-ll looksgood laibao101 mnixry coolyuantao jsjzh jhlxz2003 lqmeta littleliar liar0320 opportunityliu lubezhang lemonliu2023 kkaaddff jimmy-ajia perrychi researchmore laoboxie aldler wanmeizty disanweidu zwlijun hezhengxu2018 ahwgs wellaiyo yezhonghu0503 daddyauden jasonbrook97 harryliisme zheaoli tsdk yuanzhaok zongzmz ic- vimplus zzfn wandergis liuyq1209 yisibl ricbet magicfollower djun html1024 hcfq1999 codeyu001 chief-fei xucode2016 windhc unbyte uestccarpediem hljwkwm houyaowei lucaszhu2zgf 365-w wooodhead

cnpmcore's Issues

npm8 缺少 hasInstallScript 参数导致安装失败

原 issue：cnpm/cnpmjs.org#1667
请问这个问题在 cnpmcore 上有修复吗？
在 npm7 / 8 上都出现了跑 node-gyp 而不是 node-pre-gyp 的情况，我 curl 了一下发现也是缺少 hasInstallScript 的字段导致的。

排查过程

返回只节选最新版本的描述

registry.npmjs.org

有 hasInstallScript
curl -v https://registry.npmjs.org/xprofiler -H 'accept: application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, /' | jq | less

{
  "1.4.0": {
        "name": "xprofiler",
        "version": "1.4.0",
        "dependencies": {
          "@xprofiler/node-pre-gyp": "^1.0.9",
          "moment": "^2.29.1",
          "nan": "^2.15.0",
          "uuid": "^8.3.2",
          "yargs": "^15.4.1"
        },
        "devDependencies": {
          "@istanbuljs/schema": "^0.1.3",
          "autod": "^3.1.2",
          "clang-format": "^1.6.0",
          "codecov": "^3.8.3",
          "eslint": "^6.8.0",
          "expect.js": "^0.3.1",
          "formstream": "^1.1.1",
          "mm": "^3.2.0",
          "mocha": "^7.2.0",
          "nyc": "^15.1.0",
          "tunnel-agent": "^0.6.0",
          "urllib": "^2.38.0"
        },
        "bin": {
          "xprofctl": "bin/xprofctl"
        },
        "dist": {
          "integrity": "sha512-ZXRrsC9S602oEH6lGe0ulLSK0iZaz9aO3apxNNFtsbhrHjvifnEfUvukngTZnCDJqfcdT0RNi4x0O3x3IRtlBg==",
          "shasum": "967b4b25922fb710edfe1070e7d3aae092d1ee63",
          "tarball": "https://registry.npmjs.org/xprofiler/-/xprofiler-1.4.0.tgz",
          "fileCount": 101,
          "unpackedSize": 1185694,
          "npm-signature": "-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJiNff5ACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmrpsg/
  /WxW3nU3hMOzT2oaVWgrKrOO0JUOb3p88f827kU0d8jeQx9jL\r\nzdMI2oU4HPvGjnc1RxJ4EcoXsgSGAXfhCrRj6XiVVvzsjDniXFlAucQWouQV\r\nQVaOQj1Oq0DentvueYyuLWgrwVDexQHMbS6cFDmULd+zDedfBHRWXLSBr/9h\r\nvihImWDvsjB2QD1xt4rn22gW5DX/1PdCUsr7YfAQAE8ZxY6vucFXjWZCtZe8\r\ndBFN74W0jUdO4QoI9lY3FiNDC9SV5EWd/TG3fl4ZC/yy8tXCpNg/JM5RurjC\r\nv5J//cSxKZxbUGsZX5M5nOIsGdDt5VyjaDQoIkPeM09Apj1/Sp9Y5TjEBalE\r\nrAi1C09o7YxLUTNelI4yjsKw784w21culEPgqL+o/MEetSe5sfc/32K6zyNy\r\nCL/BYq0kNMSzDNnqOHTka2PG/sSQPszt2B8GuXEsEb0jRzcS1K3D3JV5b0b3\r\nLqMFI/1VceD09TfJMPEDJrb+xbaw9PwGB4mFCU8vH4LA5utyDYpOPNzos4cv\r\ngenIVW6qVCQqSEHZKrC3k5yTLxxRp318CgG4/dX2CG1e1h79g2WqwuGnACIm\r\n6tQufKZVdqokSyilx3dy75FpztwiAf+CdyT5OCDuOJUenJD81hNR8mKvMuQm\r\nvTLvvU/xZhdiaN6aX5QMUD4oEjP9swpAuAE=\r\n=lema\r\n-----END PGP SIGNATURE-----\r\n"
        },
        "hasInstallScript": true
      }
    }
}

registry.npmmirror.com

无 hasInstallScript
curl -v https://registry.npmmirror.com/xprofiler -H 'accept: application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, /' | jq | less

{
  "1.4.0": {
        "name": "xprofiler",
        "version": "1.4.0",
        "dependencies": {
          "@xprofiler/node-pre-gyp": "^1.0.9",
          "moment": "^2.29.1",
          "nan": "^2.15.0",
          "uuid": "^8.3.2",
          "yargs": "^15.4.1"
        },
        "devDependencies": {
          "@istanbuljs/schema": "^0.1.3",
          "autod": "^3.1.2",
          "clang-format": "^1.6.0",
          "codecov": "^3.8.3",
          "eslint": "^6.8.0",
          "expect.js": "^0.3.1",
          "formstream": "^1.1.1",
          "mm": "^3.2.0",
          "mocha": "^7.2.0",
          "nyc": "^15.1.0",
          "tunnel-agent": "^0.6.0",
          "urllib": "^2.38.0"
        },
        "bin": {
          "xprofctl": "bin/xprofctl"
        },
        "directories": {},
        "dist": {
          "integrity": "sha512-ZXRrsC9S602oEH6lGe0ulLSK0iZaz9aO3apxNNFtsbhrHjvifnEfUvukngTZnCDJqfcdT0RNi4x0O3x3IRtlBg==",
          "shasum": "967b4b25922fb710edfe1070e7d3aae092d1ee63",
          "tarball": "https://registry.npmmirror.com/xprofiler/-/xprofiler-1.4.0.tgz",
          "fileCount": 101,
          "unpackedSize": 1185694,
          "npm-signature": "-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJiNff5ACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmrpsg//WxW3nU3hMOzT2oaVWgrKrOO0JUOb3p88f827kU0d8jeQx9jL\r\nzdMI2oU4HPvGjnc1RxJ4EcoXsgSGAXfhCrRj6XiVVvzsjDniXFlAucQWouQV\r\nQVaOQj1Oq0DentvueYyuLWgrwVDexQHMbS6cFDmULd+zDedfBHRWXLSBr/9h\r\nvihImWDvsjB2QD1xt4rn22gW5DX/1PdCUsr7YfAQAE8ZxY6vucFXjWZCtZe8\r\ndBFN74W0jUdO4QoI9lY3FiNDC9SV5EWd/TG3fl4ZC/yy8tXCpNg/JM5RurjC\r\nv5J//cSxKZxbUGsZX5M5nOIsGdDt5VyjaDQoIkPeM09Apj1/Sp9Y5TjEBalE\r\nrAi1C09o7YxLUTNelI4yjsKw784w21culEPgqL+o/MEetSe5sfc/32K6zyNy\r\nCL/BYq0kNMSzDNnqOHTka2PG/sSQPszt2B8GuXEsEb0jRzcS1K3D3JV5b0b3\r\nLqMFI/1VceD09TfJMPEDJrb+xbaw9PwGB4mFCU8vH4LA5utyDYpOPNzos4cv\r\ngenIVW6qVCQqSEHZKrC3k5yTLxxRp318CgG4/dX2CG1e1h79g2WqwuGnACIm\r\n6tQufKZVdqokSyilx3dy75FpztwiAf+CdyT5OCDuOJUenJD81hNR8mKvMuQm\r\nvTLvvU/xZhdiaN6aX5QMUD4oEjP9swpAuAE=\r\n=lema\r\n-----END PGP SIGNATURE-----\r\n",
          "size": 209954
        },
        "_hasShrinkwrap": false
      }
    }
}

vega-typings 0.22.1 not found (published one month ago)

npm: https://www.npmjs.com/package/vega-typings/v/0.22.1

not found in https://npmmirror.com/package/vega-typings

need help Thanks!

bug-versions

非 cnpm cli 的客户端也能享受到 bug-versions。

之前的 https://github.com/cnpm/cnpmjs.org/pull/1683/files 似乎没迁移过来。

使用 redis 实现任务队列替换 MySQL 方案

参考 https://github.com/X-Profiler/xtransit-manager/blob/master/app.js#L13

目前的 tasks 任务读取在 tasks 表达到万行的时候性能会有问题。

支持完全不同步的私有模式

非私有模块完全使用上游 registry 的数据，需要做 302 跳转转发逻辑。

sync: all

同步所有三方包，不存在先 proxy 然后异步创建一个同步任务

sync: none

不同步任何三方包，完全 proxy

Done

Todo

GET https://registry.npmjs.org/-/npm/v1/user
npm profile get --json
{
  "tfa": {
    "pending": false,
    "mode": "auth-only"
  },
  "name": "foo",
  "email": "[email protected]",
  "email_verified": true,
  "created": "2015-02-26T01:40:54.959Z",
  "updated": "2021-12-04T17:21:28.461Z",
  "fullname": "foo",
  "twitter": "foo",
  "github": "foo"
}

GET https://registry.npmjs.org/-/v1/search?text=egg2&size=20&from=0&quality=0.65&popularity=0.98&maintenance=0.5
'npm-command': [ 'search' ],
authorization: [ 'Bearer npm_token' ]

npm search egg --json
[

{"name":"egg","scope":"unscoped","version":"2.33.1","description":"A web framework's framework for Node.js","keywords":["web","app","http","application","framework","middleware","koa","egg"],"date":"2021-12-06T16:26:46.792Z","links":{"npm":"https://www.npmjs.com/package/egg","homepage":"https://github.com/eggjs/egg","repository":"https://github.com/eggjs/egg","bugs":"https://github.com/eggjs/egg/issues"},"publisher":{"username":"fengmk2","email":"[email protected]"},"maintainers":[{"username":"eggjs-admin","email":"[email protected]"},{"username":"fengmk2","email":"[email protected]"},{"username":"atian25","email":"[email protected]"},{"username":"dead_horse","email":"[email protected]"},{"username":"popomore","email":"[email protected]"},{"username":"wanghx","email":"[email protected]"},{"username":"hyj1991","email":"[email protected]"},{"username":"thonatos","email":"[email protected]"},{"username":"killagu","email":"[email protected]"},{"username":"coolme200","email":"[email protected]"},{"username":"xadillax","email":"[email protected]"},{"username":"ngot","email":"[email protected]"},{"username":"mansonchor.zzw","email":"[email protected]"},{"username":"hubcarl","email":"[email protected]"},{"username":"shaoshuai0102","email":"[email protected]"}]}

,

{"name":"egg-mock","scope":"unscoped","version":"4.2.0","description":"mock server for egg","keywords":["egg","mock"],"date":"2021-12-17T10:44:58.662Z","links":{"npm":"https://www.npmjs.com/package/egg-mock","homepage":"https://github.com/eggjs/egg-mock","repository":"https://github.com/eggjs/egg-mock","bugs":"https://github.com/eggjs/egg/issues"},"author":{"name":"popomore","email":"[email protected]","username":"popomore"},"publisher":{"username":"fengmk2","email":"[email protected]"},"maintainers":[{"username":"eggjs-admin","email":"[email protected]"},{"username":"fengmk2","email":"[email protected]"},{"username":"atian25","email":"[email protected]"},{"username":"dead_horse","email":"[email protected]"},{"username":"popomore","email":"[email protected]"},{"username":"wanghx","email":"[email protected]"},{"username":"hyj1991","email":"[email protected]"},{"username":"thonatos","email":"[email protected]"},{"username":"killagu","email":"[email protected]"},{"username":"coolme200","email":"[email protected]"},{"username":"xadillax","email":"[email protected]"},{"username":"ngot","email":"[email protected]"},{"username":"mansonchor.zzw","email":"[email protected]"},{"username":"hubcarl","email":"[email protected]"},{"username":"shaoshuai0102","email":"[email protected]"}]}

GET http://localhost:7001/-/_view/starredByUser?key=%22cnpmcore_admin%22

在新的 web 做出来之前怎么办？

在 cnpmjs.org 增加一个远程数据的能力，web 还是走原来的方式显示，然后数据源来自新的 registry。

等新版的 web 做好再切换过去。

争取春节期间数据流量比较少，这个时候完成切换。

TODOs

cnpmjs.org 页面展示数据来源 r.cnpmjs.org
npmmirror.com 页面展示来自 registry.npmmirror.com
上面完成之后将 cnpmjs.org 页面强制跳转到 npmmirror.com 页面

  location / {
   #     include /etc/nginx/proxy_params;
   #     proxy_pass http://127.0.0.1:7002;
   #     proxy_redirect off;
       return 302 https://npmmirror.com$request_uri;
   }

支持接入三方 OAuth 注册登录模式

如 https://github.com/flowxjs/npm/blob/master/src/adapters/http/controller/extra.controller.ts#L46

Prisma: Not Found https://cdn.npmmirror.com/binaries/prisma/all_commits/22b822189f46ef0dc5c5b503368d1bee01213980/rhel-openssl-1.0.x/introspection-engine.gz

3.12.0 未同步

https://github.com/prisma/prisma/releases/tag/3.12.0

https://registry.npmmirror.com/binary.html?path=prisma/

Support migrations after the first 1.0.0 version release

https://leoric.js.org/zh/migrations

~~model.sync~~ https://www.yuque.com/leoric/blog/pphkcr#GL8QJ 不考虑使用了，因为我们会涉及字符编码的特殊设置，手动写脚本确保本地单测环境的表结构一致 https://github.com/cnpm/cnpmcore/blob/main/test/prepare.ts#L4
migrations https://leoric.js.org/zh/migrations

404 should delete package

https://r.cnpmjs.org/-/package/@nightstomp/ngx-image-gallery/syncs/61f57443c933c892e02eb708/log

AOP 实现耗时日志打印

eggjs/egg#4868

如代替 https://github.com/cnpm/cnpmcore/blob/0c56fb36ef96a47c5f31c6d16622960ef49bb6c4/app/common/decorator/AsyncTimer.ts

从 cnpm到cnpmcore怎样迁移数据库

公司部署的cnpm更换未cnpmcore部署之后，数据库迁移有没有可以参考的文档呀

引入「查询服务」

https://www.yuque.com/liberty/rf322x/gxtqnp#VxDZb

有比较多查询请求，目前混杂在 Service 和 Repository 层，有一些是直接从 Repository 读取数据，有一些是从 Service 读数据。

如果引入查询服务，可以比较干净地分离耦合。

实现 Package Version 的发布接口

API https://github.com/cnpm/cnpmjs.org/blob/master/docs/registry-api.md#publish-a-new-package

npm registry 的 version 数据结构 https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md#version

Don't sync same package at the same time

tgz 在 registry.npmjs.org 同步可能会出现文件延迟，需要到 replicate.npmjs.com 重试一次

2022-02-24 22:50:53,094 ERROR 1770189 [-/127.0.0.1/28db8e41-9581-11ec-8424-872938e39e41/2049.339ms SCHEDULE /__schedule?path=/home/admin/application/app/schedule/SyncPackageWorker.js&interval=1000&type=all] Download tarball https://registry.npmjs.org/ilab-lib/-/ilab-lib-0.0.21.tgz error: Error [DownloadNotFoundError]: Not found, status(404)

https://registry.npmjs.org/ilab-lib/-/ilab-lib-0.0.21.tgz 404 的时候通过 https://replicate.npmjs.com/ilab-lib/-/ilab-lib-0.0.21.tgz 重试一次

support speed up github

This DEPRECATED:
git clone https://github.com.cnpmjs.org/kubernetes/kubernetes.git

this is not support
git clone https://github.com.cnpmcore.org/kubernetes/kubernetes.git

should sync modified and created on npm package

https://r2.cnpmjs.org/ppp

Introduction to Hooks

https://github.com/npm/registry/tree/master/docs/hooks

支持 range 接口查询版本

对外需要跟 npm registry 沟通，看看是否可以提供新的接口，走分页。
对内，查询 db 走分页

Todos

支持 semver range 查询

sync remove version ignore publish day limit

https://r.cnpmjs.org/-/package/@loxia/mst-viewer/syncs/61fd65a99fb501a39c4bd16b/log

[2022-02-04T17:43:10.091Z] 📖 Skip remove version 0.0.413-202012311134, because it was published(2020-12-31T11:40:52.142Z) more than 7 days

tag 同步了，但是 version 不在

curl -L https://r.cnpmjs.org/-/package/@opensumi/ide-status-bar/syncs/61f1049d8f235b9e69a1a3ee/log

[2022-01-26T08:25:13.091Z] 👉👉👉👉👉 Tips: Sync cause by "http://r.cnpmjs.org/@opensumi/ide-status-bar/sync?sync_upstream=true", parent traceId: 7b5f0af0-7e81-11ec-b480-6dd2e0c7633c 👈👈👈👈👈
[2022-01-26T08:25:13.091Z] 🚧🚧🚧🚧🚧 Start sync "@opensumi/ide-status-bar" from https://registry.npmjs.org, skipDependencies: false 🚧🚧🚧🚧🚧
[2022-01-26T08:25:13.387Z] HTTP [200] content-length: 124558, timing: {"queuing":1,"dnslookup":0,"connected":0,"requestSent":1,"waiting":290,"contentDownload":296}
[2022-01-26T08:25:13.387Z] 🚧 Syncing maintainers: [{"name":"hacke2","email":"[email protected]"},{"name":"jinboker","email":"[email protected]"},{"name":"ensorrow","email":"[email protected]"},{"name":"sakuraash","email":"[email protected]"},{"name":"erha19","email":"[email protected]"},{"name":"lengthmin","email":"[email protected]"}]
[2022-01-26T08:25:13.422Z] 🚧 Syncing versions 46 => 46
[2022-01-26T08:25:13.434Z] 🟢 log: https://r.cnpmjs.org/-/package/@opensumi/ide-status-bar/syncs/61f10568a8f92fecd3b7ef68/log
[2022-01-26T08:25:13.434Z] 🟢🟢🟢🟢🟢 https://registry.npmjs.org/%40opensumi%2Fide-status-bar 🟢🟢🟢🟢🟢

Too big tgz to sync

-rw-r--r-- 1 root root  49M Dec 16 09:01 8618e5173a6031d89b2c-cardpay-cli-0.27.41.tgz
-rw-r--r-- 1 root root  49M Dec 16 09:05 91b7e1bf6299494070db-unity-renderer-1.0.15625-20211006150505.commit-3e01b3d.tgz
-rw-r--r-- 1 root root  49M Dec 16 05:01 c1a5c0e2547775151538-unity-renderer-1.0.16400-20211013180645.commit-9136b46.tgz
-rw-r--r-- 1 root root  49M Dec 16 07:52 d86f536f25dc7198cd97-unity-renderer-1.0.18125-20211029133405.commit-3d07ee8.tgz
-rw-r--r-- 1 root root  49M Dec 16 08:07 27d666df1e54dd24e0db-node-cputil-0.2.2.tgz
-rw-r--r-- 1 root root  49M Dec 16 08:06 75f68545bfa79b7d00d4-nswag-13.10.2.tgz
-rw-r--r-- 1 root root  49M Dec 16 02:35 1e9d0330a7d96a62c403-kuzzle-1.0.0.tgz
-rw-r--r-- 1 root root  49M Dec 16 08:03 1a9670e413cb8a4c1fe9-node-cputil-0.0.4.tgz
-rw-r--r-- 1 root root  49M Dec 16 03:38 a0ee7d31ebae4b36d043-unity-renderer-1.0.18243-20211101152936.commit-7ec6c91.tgz
-rw-r--r-- 1 root root  50M Dec 16 08:25 1489aa32d4deeae59cb8-node-cputil-0.0.6.tgz
-rw-r--r-- 1 root root  50M Dec 16 07:54 82e5fb73f8ccd7baac70-aumueller-server-1.0.1-b5.tgz
-rw-r--r-- 1 root root  50M Dec 16 09:03 c47a63765a67c6c4fc16-monofo-2.1.1.tgz
-rw-r--r-- 1 root root  50M Dec 16 08:15 46fabd85cb7197241ef8-node-cputil-0.0.11.tgz
-rw-r--r-- 1 root root  50M Dec 16 07:36 0ac8ef52610070d413a1-potree-zen-1.8.1.tgz
-rw-r--r-- 1 root root  50M Dec 16 09:07 945169999a39a825e012-cardpay-cli-0.27.41.tgz
-rw-r--r-- 1 root root  50M Dec 16 08:19 b3be4b9e690a6f287f7f-node-cputil-0.0.9.tgz
-rw-r--r-- 1 root root  51M Dec 16 08:57 e362dc329bd7da2864c9-node-cputil-0.0.9.tgz
-rw-r--r-- 1 root root  51M Dec 16 08:52 172ffdf475158abc4985-nswag-13.2.0.tgz
-rw-r--r-- 1 root root  52M Dec 16 08:18 06205ceee18355409e96-node-cputil-0.0.5.tgz
-rw-r--r-- 1 root root  52M Dec 16 08:55 0f974cc056f13146a248-react-native-camera-kit-6.2.5.tgz
-rw-r--r-- 1 root root  53M Dec 16 08:13 c0973d1207ea01f16380-node-cputil-0.0.12.tgz
-rw-r--r-- 1 root root  53M Dec 16 08:12 76de61f7f07a5e3581df-node-cputil-0.0.8.tgz
-rw-r--r-- 1 root root  53M Dec 16 07:05 aa424cde4619f098f3c0-react-devtools-inline-4.17.0.tgz
-rw-r--r-- 1 root root  54M Dec 16 08:25 ef8eedd21f8c7339e8f2-node-cputil-0.0.1.tgz
-rw-r--r-- 1 root root  54M Dec 16 08:14 a8123d0f1f2371e4d2d2-node-cputil-0.0.7.tgz
-rw-r--r-- 1 root root  54M Dec 16 07:28 dea8abdfbe341d1545b7-cardpay-cli-0.27.41.tgz
-rw-r--r-- 1 root root  54M Dec 16 07:32 fe7785f5e96682a45d45-potree-zen-1.8.4.tgz
-rw-r--r-- 1 root root  54M Dec 16 08:17 2a90d1fa999823a1152d-node-cputil-0.0.10.tgz
-rw-r--r-- 1 root root  54M Dec 16 07:30 f2cab71406c17fc96577-potree-zen-1.8.5.tgz
-rw-r--r-- 1 root root  54M Dec 16 08:53 ee37aa88d4074625e52f-node-cputil-0.0.11.tgz
-rw-r--r-- 1 root root  55M Dec 16 08:59 0d7943ab798d23497547-node-cputil-0.0.6.tgz
-rw-r--r-- 1 root root  56M Dec 16 01:25 dd3adae3b6c4dcb71ebc-selenium-standalone-4.6.0.tgz
-rw-r--r-- 1 root root  56M Dec 16 07:36 cffd7a80c0414ded9ca7-aumueller-server-1.0.1-b15.tgz
-rw-r--r-- 1 root root  56M Dec 16 08:11 5a582201d9c4b7a4b495-node-cputil-0.2.0.tgz
-rw-r--r-- 1 root root  60M Dec 16 01:25 b474b932985368b34e3b-phantomjs-bin-2.1.1.tgz

精简包信息返回需要增加 libc 字段

cnpm/npminstall#386

目前只返回了 os、cpu，还需要 libc 字段。

Error: ER_DATA_TOO_LONG: Data too long for column 'name' at row 1

2021-12-14 22:50:16,434 ERROR 92622 [Job#163949322173616956023029765728591] /home/admin/application/app/schedule/SyncPackageWorker.js execute failed, used 194696ms. Error: ER_DATA_TOO_LONG: Data too long for column 'name' at row 1
    at Query.Sequence._packetToError (/home/admin/application/node_modules/[email protected]@mysql/lib/protocol/sequences/Sequence.js:47:14)
    at Query.ErrorPacket (/home/admin/application/node_modules/[email protected]@mysql/lib/protocol/sequences/Query.js:79:18)
    at Protocol._parsePacket (/home/admin/application/node_modules/[email protected]@mysql/lib/protocol/Protocol.js:291:23)
    at Parser._parsePacket (/home/admin/application/node_modules/[email protected]@mysql/lib/protocol/Parser.js:433:10)
    at Parser.write (/home/admin/application/node_modules/[email protected]@mysql/lib/protocol/Parser.js:43:10)
    at Protocol.write (/home/admin/application/node_modules/[email protected]@mysql/lib/protocol/Protocol.js:38:16)
    at Socket.<anonymous> (/home/admin/application/node_modules/[email protected]@mysql/lib/Connection.js:88:28)
    at Socket.<anonymous> (/home/admin/application/node_modules/[email protected]@mysql/lib/Connection.js:526:10)
    at Socket.emit (node:events:390:28)
    at addChunk (node:internal/streams/readable:315:12)
    --------------------
    at Protocol._enqueue (/home/admin/application/node_modules/[email protected]@mysql/lib/protocol/Protocol.js:144:48)
    at PoolConnection.query (/home/admin/application/node_modules/[email protected]@mysql/lib/Connection.js:198:25)
    at /home/admin/application/node_modules/[email protected]@leoric/src/drivers/mysql/index.js:81:18
    at new Promise (<anonymous>)
    at MysqlDriver.query (/home/admin/application/node_modules/[email protected]@leoric/src/drivers/mysql/index.js:80:21)
    at Spell.ignite (/home/admin/application/node_modules/[email protected]@leoric/src/spell.js:434:37)
    at Spell.then (/home/admin/application/node_modules/[email protected]@leoric/src/spell.js:451:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  code: 'ER_DATA_TOO_LONG',
  errno: 1406,
  sqlMessage: "Data too long for column 'name' at row 1",
  sqlState: '22001',
  index: 0,
  sql: "INSERT INTO `dists` (`gmt_create`, `gmt_modified`, `dist_id`, `name`, `path`, `size`, `shasum`, `integrity`) VALUES ('2021-12-14 22:50:16.377', '2021-12-14 22:50:16.377', '61b8af28489aa269cfabad58', 'solid-client-0.0.2-dependabotnpmandyarnwebsitedocusauruspreset-classic-200-alpha61-192892303-618.0.tgz', '/packages/@inrupt/solid-client/0.0.2-dependabotnpmandyarnwebsitedocusauruspreset-classic-200-alpha61-192892303-618.0/solid-client-0.0.2-dependabotnpmandyarnwebsitedocusauruspreset-classic-200-alpha61-192892303-618.0.tgz', 298974, '716ede0ed0df5d248fff22f8ff598e7fd49a75ce', 'sha512-pkAx/F0mtrSyfNTNgLKSgUN3hchxZi5Iz0RD8Wec6sNgwydC6YI3nXFWcYRB9hdPP5jhmiaBtEVc3DkYYtNEZQ==')"

INSERT INTO `dists` (`gmt_create`, `gmt_modified`, `dist_id`, `name`, `path`, `size`, `shasum`, `integrity`) 
VALUES ('2021-12-14 22:50:16.377', '2021-12-14 22:50:16.377', '61b8af28489aa269cfabad58', 'solid-client-0.0.2-dependabotnpmandyarnwebsitedocusauruspreset-classic-200-alpha61-192892303-618.0.tgz', 
'/packages/@inrupt/solid-client/0.0.2-dependabotnpmandyarnwebsitedocusauruspreset-classic-200-alpha61-192892303-618.0/solid-client-0.0.2-dependabotnpmandyarnwebsitedocusauruspreset-classic-200-alpha61-192892303-618.0.tgz', 
298974, 
'716ede0ed0df5d248fff22f8ff598e7fd49a75ce', 'sha512-pkAx/F0mtrSyfNTNgLKSgUN3hchxZi5Iz0RD8Wec6sNgwydC6YI3nXFWcYRB9hdPP5jhmiaBtEVc3DkYYtNEZQ==')

Binary dist index page

Maybe just using https://chromedriver.storage.googleapis.com/index.html?path= html and javascript

avoid private keyword

https://zhuanlan.zhihu.com/p/460581100?utm_source=ZHShareTargetIDMore&utm_medium=social&utm_oi=26912072138752

增加 install-node 机制，确保构建的 node 版本和实际运行的一致

这样也可以确保我们及时升级到合适的 node 版本而不需要长期维护多个 node 版本的支持。

link to socket.dev

like https://socket.dev/npm/package/egg

no maintainers data on sync package

curl -L https://r.cnpmjs.org/-/package/postman-jsdoc-theme/syncs/61f406302d73dab1352cf63b/log
[2022-01-28T15:05:20.208Z] 👉👉👉👉👉 Tips: Sync cause by "http://r.cnpmjs.org/postman-jsdoc-theme/sync?sync_upstream=true", parent traceId: b5b34510-804b-11ec-9ac5-e1526389ecca 👈👈👈👈👈
[2022-01-28T15:05:20.208Z] 🚧🚧🚧🚧🚧 Start sync "postman-jsdoc-theme" from https://registry.npmjs.org, skipDependencies: false, syncDownloadData: false, attempts: 1 🚧🚧🚧🚧🚧
[2022-01-28T15:05:21.063Z] HTTP [200] content-length: undefined, timing: {"queuing":1,"dnslookup":1,"connected":4,"requestSent":9,"waiting":854,"contentDownload":854}
[2022-01-28T15:05:21.063Z] ❌ invalid maintainers: [], log: https://r.cnpmjs.org/-/package/postman-jsdoc-theme/syncs/61f406302d73dab1352cf63b/log
[2022-01-28T15:05:21.063Z] ❌❌❌❌❌ https://registry.npmjs.org/postman-jsdoc-theme?t=1643382320209 ❌❌❌❌❌

同步时机可能需要更加精细化

curl https://r2.cnpmjs.org/-/package/ms/syncs/61b6eb2c686532287ed4484c/log -L
版本太多，目前每次同步一个版本都会去刷新全量的 list 到 oss，导致每个版本都要十几秒才能执行完。

DEVELOPER. MD has a word wrong

并行同步冲突

2021-12-18 14:24:59,667 INFO 132658 [-/127.0.0.1/0a00003916398086996626952132658/6.815ms SCHEDULE /__schedule?path=/home/admin/application/app/schedule/SyncPackageWorker.js&interval=1000&type=all] [SyncPackageWorker:subscribe:executeTask:start][2] taskId: 61bd7ebbe9de14062b37d90f, targetName: vue-manage-system-dev, attempts: 1, params: {"tips":""}, updatedAt: 2021-12-18T06:24:59.663Z, delay -1ms
2021-12-18 14:25:19,148 INFO 132658 [-/127.0.0.1/0a00003916398086996626952132658/19488.093ms SCHEDULE /__schedule?path=/home/admin/application/app/schedule/SyncPackageWorker.js&interval=1000&type=all] [SyncPackageWorker:subscribe:executeTask:success][3] taskId: 61bd7ebbe9de14062b37d90f, targetName: vue-manage-system-dev, use 19486ms

2021-12-18 14:25:04,027 INFO 46000 [-/127.0.0.1/0a00003a1639808704020682246000/8.322ms SCHEDULE /__schedule?path=/home/admin/application/app/schedule/SyncPackageWorker.js&interval=1000&type=all] [SyncPackageWorker:subscribe:executeTask:start][2] taskId: 61bd7ebfc387e40632a4e85f, targetName: vue-manage-system-dev, attempts: 1, params: {"tips":""}, updatedAt: 2021-12-18T06:25:04.022Z, delay -2ms
2021-12-18 14:25:18,515 ERROR 46000 [-/127.0.0.1/0a00003a1639808704020682246000/14496.475ms SCHEDULE /__schedule?path=/home/admin/application/app/schedule/SyncPackageWorker.js&interval=1000&type=all] [SyncPackageWorker:subscribe:executeTask:error][1] taskId: 61bd7ebfc387e40632a4e85f, targetName: vue-manage-system-dev, Error: ER_DUP_ENTRY: Duplicate entry '/packages/vue-manage-system-dev/full_manifests.json' for key 'uk_path'

Support OpenTelmetry instead of opentracing

https://opentelemetry.io/

Support changes_stream on cnpmcore

https://github.com/npm/registry-follower-tutorial

{ seq: 445,
  id: 'CompoundSignal',
  changes: [ { rev: '5-a0695c30fdaa3471246ef0cd6c8a476d' } ] }
{ seq: 446,
  id: 'amphibian',
  changes: [ { rev: '5-1a864e76d844e90bf6c63cb94303b593' } ] }
{ seq: 447,
  id: 'aop',
  changes: [ { rev: '9-9acc0139df57a1db2604f13f12b500f2' } ] }
{ seq: 448,
  id: 'dynamo-schema',
  changes: [ { rev: '5-bf8052c0d4b6e80e6664625137efd610' } ] }
{ seq: 451,
  id: 'password-reset',
  changes: [ { rev: '21-948e6633799ffd56a993c3fb144d1728' } ] }

sync user

https://npmmirror.com/sync/user:acdlite?spm=a2c6h.24755359.0.0.250c535dMBYBv9#logid=75183887

同步原来的 downloads 数据

主要是避免单个包的历史下载数据丢失问题，全站数据我们已经有统计过。

核心逻辑

设定某一天是重合期，~~这一天的下载量相加~~，这一天的下载量是覆盖。这一天之后的数据忽略。

选择一个周日（反正从0点开始丢弃的数据也能被接受）开始跑，那么这一天的数据允许丢弃老 registry 的数据，因为可能会重跑多次，这样避免数据做相加导致额外增多了，宁可丢掉重合那天的数据，也不要制造多出来的下载量。

Todos

r.cnpmjs.org datas 选择 2021-12-28
registry.npmmirror.com datas 选择 2022-01-29

Sync removed packages

Some packages was removed by npm and response 404 status
https://registry.npmjs.com/@ktrace/web

Allow CORS

https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS

这样后续 Web 应用可以前后端分离，任何前端都可以基于 registry api 做他的个性化 Web。

CORS
user token 支持 JSONWebToken，这样用户认证也支持了。https://jwt.io/introduction

cnpm / cnpmcore Goto Github PK

cnpmcore's Introduction

cnpm

Requirements

Install

Usage

Sync packages from npm

Open package document or git web url

Build your own private registry npm cli

Install with original npm cli

License

Contributors

cnpmcore's People

Contributors

Stargazers

Watchers

Forkers

cnpmcore's Issues

排查过程

registry.npmjs.org

registry.npmmirror.com

sync: all

sync: none

Done

Todo

TODOs

Todos

核心逻辑

Todos

Recommend Projects

Recommend Topics

Recommend Org

Sync packages from `npm`