gazmull / kh-snek Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 1.0 832 KB

RESURRECTION IN PROGRESS - [R18] Dedicated Assets Tool for Kamihime DB.

Home Page: https://kamihimedb.win

License: MIT License

JavaScript 1.66% TypeScript 98.34%

kh-snek's People

Contributors

Stargazers

Watchers

Forkers

sariohara

kh-snek's Issues

Very Balanced Worker Spawn Method

The CPU cores are wasted due to the current code only distributing Resources per character (which is 3 Resources on average). On average, only 3 CPU cores were being occupied by the DownloadManager. Currently using the latest commit on a 12-core cloud instance and I'm wasting cents each second passes by.

Currently I can only propose that to occupy all CPU cores: do a round-robin distribution on Character[] as well. For of is very weak in this case so try to avoid it.

For Reference:

kh-snek/src/lib/DownloadManager/index.ts

Lines 43 to 59 in bfdb209

    
           for (const char of this.data as ICharacter[]) { 
        
             const resources = [ ...char.resources.entries() ]; 
        
             const balancedData = this._balanceData<[hashIdentifier, IResourceValues]>(resources); 
        
             const workers = await Promise.all( 
        
               balancedData.map((val, idx) => { 
        
                 const finalData: ICharacter = { 
        
                   id: char.id, 
        
                   name: char.name, 
        
                   resources: new Collection(val) 
        
                 }; 
        
                 return this._spawnWorker(idx, finalData); 
        
               }) 
        
             ); 
        
             workersResult.push(...workers); 
        
           }

Remove Scenarios folder destination dependency with Scripts folder

Feat: Add Flag For EX Episodes

Not sure how this exactly works at the moment but current solution is look up by incremental values (e.g. once 4 EX episodes were extracted, and another batch got released, resume from 5th EX episode being ++4).

Incorrect directory (files.rsc)

The last folder should be the episode's hash instead of DB's column name. What a massive blunder, but if this were to be considered a patch, the user will have to re-download all thousands of files again (Mine took 8 hours @ 100Mbps) unless I ignore existing files which is not really recommended in my part yet. It's not really a major breaking oversight, but just annoying as a structure overview. This will be considered for the next major version.

kh-snek/src/lib/DownloadManager/worker.ts

Line 154 in f13daab

const fileNamesPath = `${auth.destinations.scenarios}${char.id}/${key}/`;

Feat: Make Resource Extractor Multi-Threaded

The only multi-threaded section of this project is the "Download Manager." Extractor could benefit as well specially on operations that start from scratch which takes too long to be done in the current single-thread design.

Start from here:

kh-snek/src/lib/Extractor.ts

Line 71 in c2daed9

this.resourcesExtracted += await this._extract(id, resources);

Feat: Ability to skip doing webp process

— with process argument -nw or --nowebp. Webp has no real use yet on KamihimeDB's Device.

gazmull / kh-snek Goto Github PK

kh-snek's People

Contributors

Stargazers

Watchers

Forkers

kh-snek's Issues

Very Balanced Worker Spawn Method

Remove Scenarios folder destination dependency with Scripts folder

Feat: Add Flag For EX Episodes

Incorrect directory (files.rsc)

Feat: Make Resource Extractor Multi-Threaded

Feat: Ability to skip doing webp process

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	for (const char of this.data as ICharacter[]) {
	const resources = [ ...char.resources.entries() ];
	const balancedData = this._balanceData<[hashIdentifier, IResourceValues]>(resources);
	const workers = await Promise.all(
	balancedData.map((val, idx) => {
	const finalData: ICharacter = {
	id: char.id,
	name: char.name,
	resources: new Collection(val)
	};

	return this._spawnWorker(idx, finalData);
	})
	);

	workersResult.push(...workers);
	}