pashpashpash / manga-reader Goto Github PK

This is a more general task, but it's important to clean up the code. Right now it's a bit of a mess with python objects being created with unstandardized parameters (for example, the volume object, narration_script, movie_script object, etc. These should likely be classes instead of objects.

The goal here is for the source code to be cleanly organized into understandable, standardized objects/classes/functions so that even people with a limited understanding of code can feel like they can modify the behavior of the code to their liking. This will also help other developers get involved and understand what the code does.

Lastly, the README should be updated to include all of the setup steps, as right now certain things are excluded (i.e. needing to set up torch among other libraries prior to running)

Looking forward, the AI-generated videos SORA model looks promising. The pieces that we have built now can be used to turn MangaRecap into a full fledged animation studio. Input a manga and it creates an entire animated recap for you, with accurate characters and plots.

Right now, the script feeds every single page of a manga volume into GPT vision to identify chapter start pages in order to split up the volume into chapters later in the code. This happens as part of the "identifying important_pages" step, which is slow and expensive. There are better ways of doing this including but not limited to identifying the table of contents and mapping the table of contents chapter pages to the relevant PDF page indexes.

https://github.com/pashpashpash/manga-reader/blob/main/app.py#L39-L66

Right now users have to create a "profile reference" PDF that has an example profile reference page for the manga. Then as part of the "identify important_pages" step, GPT vision is used to identify a character profile page within the volume. This is far from ideal, as it slows down the time it takes for a user to run the script both in terms of extra setup and slow GPT vision processing time. This is far from ideal.

Perhaps this can help:
https://github.com/ragavsachdeva/magi

I'm happy for other ideas on how to improve this.

I have had some people tell me that the GPT concurrent calls I am making right now are being throttled -- most likely because my Openai organization has higher concurrency limits compared to new accounts. Retries should be implemented as well.

https://github.com/pashpashpash/manga-reader/blob/main/app.py#L47-L66

https://github.com/pashpashpash/manga-reader/blob/main/app.py#L173-L185

From someone who attempted to run the code:

openai.RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4-vision-preview in organization org-xxxxxxxxxx on tokens per min (TPM): Limit 10000, Requested 25012. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

Perhaps we can use the same tech these guys use? https://toona.io/colorizer

There's lots of room to optimize the processing costs, including switching away from elevenlabs to a cheaper azure TTS for example.

Picrel, many people have expressed interest in this feature. cc @rishi23root

There's also a lot that can be improved with the quality of the videos. There can be customizeable backgrounds, panning, zooming into panels, and more creative panel/page placement.

This is a good benchmark for video quality: https://www.youtube.com/watch?v=CrYApEUR904&t=14599s

https://github.com/Zehina/Webtoon-Downloader