Get chapter list using WordPress API about webtoepub HOT 2 CLOSED

@Mathnerd314
It's an interesting idea. However, I have to veto the idea of providing code snippets that are run. Google and Mozilla do NOT like use of exec() in extensions, as it's too difficult to automatically verify that you're not doing something that violates their TOS.

And to be perfectly honest, I'd be surprised if > 2% of the people who use WebToEpub would be able to write scripts. So, would add more complication that is not useful to nearly all users.

Adding a button to fetch the posts that only works for Wordpress hosted sites, seems like a significant amount of work for a small gain. But if you're willing to do a Pull Request, I'll certainly consider it.

I did try wiring fetching the posts into the Wordpress base parser.

    async getChapterUrls(dom) {
        let content = this.findContent(dom).cloneNode(true);
        this.removeUnwantedElementsFromContentElement(content);
        let list = util.hyperlinksToChapterList(content);
        if (0 < list.length) {
            return list;
        }
        return this.getChapterUrlsViaWordpressAPI(dom);
    }

    async getChapterUrlsViaWordpressAPI(dom) {
        let host = new URL(dom.baseURI).hostname;
        if (!host.includes("wordpress")) {
            return [];
        }
        let chapters = [];
        let page = 1;
        let data = null;
        do {
            let url = `https://public-api.wordpress.com/wp/v2/sites/${host}/posts?page=${page}`;
            let res = await fetch(url);
            data = await res.json();
            for(let result of data) {
                let chapter = ({
                    sourceUrl:  result.link,
                    title: result.title.rendered,
                });
                chapters.push(chapter);
            }
            page++;
        } while(data.length >= 10)
        return chapters;
    }

Unfortunately, it's not the best. It's slow and is a mess when the chapters are in same order as posts, (Which seems to happen often.)

Finally, I'm curious what how you

I ran this JS in the context of the site

Could you provide more detail, please?

For my notes: 37 minutes work.

from webtoepub.

I ran this JS in the context of the site

Could you provide more detail, please?

I just opened the developer console, pasted it in, and ran it. (after disabling the stupid no-paste protection)

I have to veto the idea of providing code snippets that are run. Google and Mozilla do NOT like use of exec() in extensions, as it's too difficult to automatically verify that you're not doing something that violates their TOS.

Fair, getting past the extension review is definitely a concern. Although, there is a userScript API designed for that purpose. It looks like you can define a callback processPages or something, expose it as a global, and then you could implement each site/page processor as a user script that examines the page and calls processPages. So it wouldn't call exec. A lot of work though, and as you say most users won't be writing scripts. For my purposes it is probably easier to maintain a local fork of the extension and open PR's as I implement site fixes.

Adding a button to fetch the posts that only works for Wordpress hosted sites, seems like a significant amount of work for a small gain. But if you're willing to do a Pull Request, I'll certainly consider it.

Yeah I'm not sure about how often this is needed, I really was just posting the snippet here so I didn't lose track of it. That part is done but if I end up needing this functionality again I'll open a PR. I do think it should be manual and in the chapter list UI, it is too slow and requires too much chapter munging to use as the base fetcher.

from webtoepub.