obetomuniz / tatooine Goto Github PK

Tatooine Plugins unlock the library for anyone that wants to extend the current library capabilities. The current plugin type supported is transformer. This type of plugin allows pre and post data processing.

Soon,engine plugins will be supported to allow developers to add their scraping engines alongside Tatooine capabilities.

To-Do

• Document Plugins Creation and Usage
• #365

Extra Considerations

Tatooine Conditions

Tatooine Conditions will now live as a Tatooine Plugin. Also, its codebase was already removed from the library defaults.

Soon Tatooine Conditions Plugin will be moved to another repository. For now, use it like this:

import { scrape, conditionsPlugin as conditions } from "tatooine"

await scrape({
  url: "https://betomuniz.com",
  engine: "html",
  options: {
    selectors: {
      title: {
        selector: ".cDZeRL li a",
      },
    },
    plugins: [
      conditions({
        title: [
          {
            operation: "regex",
            value: "^How",
            sensitive: false,
          },
        ],
      }),
    ],
  },
})

Add condition to help developers filter results on every method based on: ${selector} ${operation} ${value}.

selector: Stands for the value to be filtered
operation: Stands for ==, = and != initially
value: Stands for the value to use for filtering

Dependabot encountered the following error when parsing your .dependabot/config.yml:

The property '#/update_configs/0/allowed_updates/0/match/update_type' value "direct" did not match one of the following values: all, security

Please update the config file to conform with Dependabot's specification using our docs and online validator.

This project is great! Thanks a ton for doing this. It's well thought out and very flexible.

One thing I'd love to see is some TS definitions. I'm happy to add them following the schema engine docs, if you're open to it.

It can be useful for users that need to Authenticate or even Preprocess the content before crawling the <markup />.

On the other hand, it can bring extra complexity in the API proposal. Leaving it here for afterward thoughts.

• Define an easy API for HTTP headers and auth
• Define defaults for HTTP headers
• Define defaults based on the scrape method being used
• Define defaults based on best practices for performance, security, and privacy

Hi @obetomuniz, first congratulations on the project. I would like to suggest an improvement in the documentation of the examples.

Perhaps an HTML page or in the README with clearer examples will be better. I was a little confused about how to use Tatooine.

For example:

Tatooine([rss]).then(...)

What do I need to do here?

parse is meaningful, but doesn't cover the real goal with the functions.

Same for standalone functions as parseEngineX > scrapeEngineX

Add compatibility with Deno runtime is considered. But first Deno should be stable, well adopted by the community, and be supported in the NPM.

This issue will cover the updates of the tatooine v2

Must Have:

• Use rollup to packaging
• Use axios instead of request
• Eslint & Prettier
• Jest
• Travis CLI
• Codecov
• Create a new API architecture with a focus on a clean and easy way to plug schemas/sources with the Tattooine Scraper engine

This issue will be updated if new stuff appears or it finishes.

This will allow developers to use tatooine only for data processing

https://github.com/obetomuniz/tatooine/blob/master/src/engines/spa.js#L9-L26 should be covered properly.