anish-agnihotri / flipper Goto Github PK

View Code? Open in Web Editor NEW

80.0 2.0 16.0 215 KB

Scrape ERC721 metadata, flip images, test creative permissiveness.

TypeScript 100.00%

nft flipping phunk phayc

flipper's Introduction

Flipper

Allows scraping ERC721 tokens for their metadata and images, flipping images and uploading new metadata to IPFS.

Metadata retrieval:

Supports both IPFS and centralized metadata stores
Supports retrieving and storing images from IPFS and centralized sources in source quality
You can stop Flipper at any time and pick back up where you left off, in the future.

Usage

# Install dependencies
npm install

# Copy environment variables
cp .env.sample .env

# Edit environment variables
vim .env

# Run flipper
npm run start

IPFS

A majority of Ethereum NFT metadata is stored via IPFS. While you can use a public IPFS gateway with these scripts, you will be quickly rate-limited.

As such, it is recommended to run your own local node (<1m setup time):

# Install ipfs
# Mac:
brew install ipfs
# Debian
apt install ipfs -y

# Initialize
ipfs init

# Start daemon
ipfs daemon

Or, to get a private gateway from a provider like Pinata ($20/month).

Flipped metadata is also stored on IPFS. It is recommended to use a gateway from a pinning service, so you do not lose your metadata upon killing your local daemon.

Potential expansions

Support retrieving metadata and images from Arweave (comply with ar:// standard)
Support storing flipped metadata and images on Arweave (bulk upload w/ private key)
Parallelize retrieval (fast enough to scrape ~10K collection in <5m with local IPFS node + erigon, but can be faster with hosted nodes)

flipper's People

Contributors

Stargazers

Watchers

Forkers

nft-standards-wg hey-ynot h0rizons wertercatt federicosan delta0800 liulinhui funkymonkey5150 nur-yalwa m1keyc vbxprotocol-development svalkaforkov gsknnft bedlam520 rishabh-gupta2 daytahub

flipper's Issues

IPFS urls are incorrectly converted.

[wertercatt@RGBeast flipper]$ cat .env
IPFS="http://127.0.0.1:8080/ipfs/"
RPC="https://mainnet.infura.io/v3/[secret-removed]"
CONTRACT="0x3b3ee1931dc30c1957379fac9aba94d1c48a5405"

# Optional for automatic uploading of metadata (requires pinFileToIPFS perm)
PINATA_JWT=""

When given the tokenURI 'https://ipfs.foundation.app/ipfs/QmXBC4K9SiZaqbRMEpCaDu8322gVLj5fKRiDTvbziJh62N/metadata.json' the script saves the contents of '$IPFS/QmXBC4K9SiZaqbRMEpCaDu8322gVLj5fKRiDTvbziJh62N' to the token JSON file (resulting in it just containing the HTML directory page) instead saving the contents of "$IPFS/QmXBC4K9SiZaqbRMEpCaDu8322gVLj5fKRiDTvbziJh62N/metadata.json' to the file as it should.

Script fails after downloading 10-15 items

Use async file I/O

Summary

The current implementation uses the await fs.writeFileSync . This leads to reduced performance for the script with contention on file I/O.

Suggestion

Wanted to understand if there is any particular reason you chose to go with sync file I/O, instead of async implementation writeFile. If not, I can raise a PR for the same, let me know.

Script crashes if a token number doesn't exist

[wertercatt@RGBeast flipper]$ npm run start

> [email protected] start
> ts-node src/app.ts

info: Scraping Foundation collection (supply: 117894)
info: original metadata folder exists till token #39
info: flipped metadata folder exists but is empty
/home/wertercatt/flipper/node_modules/@ethersproject/logger/src.ts/index.ts:225
        const error: any = new Error(message);
                           ^
Error: call revert exception (method="tokenURI(uint256)", errorArgs=["ERC721Metadata: URI query for nonexistent token"], errorName="Error", errorSignature="Error(string)", reason="ERC721Metadata: URI query for nonexistent token", code=CALL_EXCEPTION, version=abi/5.5.0)
    at Logger.makeError (/home/wertercatt/flipper/node_modules/@ethersproject/logger/src.ts/index.ts:225:28)
    at Logger.throwError (/home/wertercatt/flipper/node_modules/@ethersproject/logger/src.ts/index.ts:237:20)
    at Interface.decodeFunctionResult (/home/wertercatt/flipper/node_modules/@ethersproject/abi/src.ts/interface.ts:425:23)
    at Contract.<anonymous> (/home/wertercatt/flipper/node_modules/@ethersproject/contracts/src.ts/index.ts:392:44)
    at step (/home/wertercatt/flipper/node_modules/@ethersproject/contracts/lib/index.js:48:23)
    at Object.next (/home/wertercatt/flipper/node_modules/@ethersproject/contracts/lib/index.js:29:53)
    at fulfilled (/home/wertercatt/flipper/node_modules/@ethersproject/contracts/lib/index.js:20:58)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  reason: 'ERC721Metadata: URI query for nonexistent token',
  code: 'CALL_EXCEPTION',
  method: 'tokenURI(uint256)',
  errorArgs: [ 'ERC721Metadata: URI query for nonexistent token' ],
  errorName: 'Error',
  errorSignature: 'Error(string)',
  address: '0x3b3ee1931dc30c1957379fac9aba94d1c48a5405',
  args: [ 40 ],
  transaction: {
    data: '0xc87b56dd0000000000000000000000000000000000000000000000000000000000000028',
    to: '0x3B3ee1931Dc30C1957379FAc9aba94D1C48a5405'
  }
}
[wertercatt@RGBeast flipper]$ cat .env
IPFS="http://127.0.0.1:8080/ipfs/"
RPC="https://mainnet.infura.io/v3/[secret-removed]"
CONTRACT="0x3b3ee1931dc30c1957379fac9aba94d1c48a5405"

# Optional for automatic uploading of metadata (requires pinFileToIPFS perm)
PINATA_JWT=""

anish-agnihotri / flipper Goto Github PK

flipper's Introduction

Flipper

Usage

IPFS

Potential expansions

flipper's People

Contributors

Stargazers

Watchers

Forkers

flipper's Issues

IPFS urls are incorrectly converted.

Script fails after downloading 10-15 items

Use async file I/O

Summary

Suggestion

Script crashes if a token number doesn't exist

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent