redco / goose-parser Goto Github PK

A project with configured build for browser env and working basic usage example. Users will be able to clone this repo and start hacking!

For example we need to do two actions:

• parse URL
• parse Id from URL (to simplify that action we should use results from first one).

Add ability to add custom pagination method

Sometime we need to specify data type in the parsed result.
For now this is a "type" field. But also we use "type" keyword as determine type of actions and transforms . So we need to rename "type" to "dataType" in cases when it has data type meaning.

Now when we have plenty of docs, it's time to add a table of content in the head of Readme.md

Add docker files to use goose-parser from envs:

• phantom-environment
• chrome-environment
• jsdom-environment

Sometimes we need to click or do some custom actions before parse a row from grid.

Example to specify event:
{event: 'parse.pre', scope: 'div.expl-open-ticket-button', action: 'click'}

Add selenium environment to simplify testing process

We have stores now and can set data in one action and get it inside another, useActionsResult is used to get actions result from previous actions, it's rudiment and should be removed

Ability to go deep to row information page to parse it inside.
For example, we have list of rows, but each of it has information page, where we should parse the row.

Needed:

• add some DataTransformManager.
• add several DefaultTransformers like date, split, ...
• ability to add custom transformers.
• in the parse nodes should be added and ability to specify chains of transformers, which will be applied one by one to the parsed data.

Example:
We have parsed data:

• date: "13 February 2016, Sat"

We need to get:

• date: "13 February 2016 13:35" or even "2015-02-13 13:35:00"

Return new parsing scope from action

It should:

• knows how to detect if captcha had happened (unexpected landing to captcha url or existing on the page captcha scope)
• knows scope of confirm button and input for enter captcha.
• take a picture of captcha (do base64 from picture in evaluate)
• has ability to set a promised function, which will determine captcha and return value.

Implement:

• does action back (if needed) and continue parsing

Add documentation

Ability to parse scope attributes. For example row deep link.

Add and configure coveralls.io

Prepare basic documentation about

• actions and a way to add custom actions
• transforms and a way to add custom transforms
• cookie storage
• langoose

For now on PhantomError we get only error, but this could happen after parsing several rows.

• #54.
• we need to provide Parser scope to restore it.
• we need to have an ability to continue parsing process after an error, from the place when it had stopped.

Ability to parse or set row identifier depends on row scope.
By this identifier we can say that this row is unique among others.

Add tests

Needed:

• BrowserEnvironment
• Script to pack dependencies to one js file

When page load pagination happens, application getting errors about Sizzle (because promise trying to check data with it).

For now on PhantomError we get only error, but this could happen after parsing several rows.
We need to provide results together with error.

Before parse we need to check if that row required to be parsed.
For example we have list of ids previously parser, compare if current row _id is in the list - continue without parsing.
!! We need to parse _id first before parsing all others.

We can extend rules and actions from any defined one in the system. And have an ability to override some particular properties.

This will allow to maintain a big amount of similar rules in easy way.

PhantomEnv should allow to set proxies list and knows moment when to switch it between each others.
Remember which proxies was used and last time and url of using.
Probably, also set strategy of switching:

• each query,
• smart query (emulate user action).

Add ability to paginate via simple pages

Ability to parser URL in simple rule

Add to docs:

• Describe Environments.
• Describe dependency to PhantomJS 2.0.
• Describe process of tests for different environments.
• Add budgets from shields.io.
• Describe how to debug parser.

Needed:

• ability to extend actions with your own
• ability to extend transformations with your own

Needed:

• Ability to specify:
  • schema from
  • schema to
  • DataTransformers (#2) which is needed to apply during transformation

Example:
We have parsed data:

• time: "13:35"
• date: "13 February 2016, Sat"

We need to get:

• date: "13 February 2016 13:35" or even "2015-02-13 13:35:00"

Add SlimerEnvironment
https://slimerjs.org

#77
#78
#69

• Add to envs ability to save and restore Parser scopes.
• Fix tests for BrowserEnv for waitForPage

Add excludes for loading resources on the page - attach to event which allows to cancel loading.

That kind is very close to scroll pagination, but instead of scrolling you need to click on the block to load new page [extend current page list].

Need to think about a way to build custom pagination event, which allows to cover any case of pagination.

import {
   PhantomEnvironment,
   Parser
} from 'goose-parser';

const env = new PhantomEnvironment({
   url: 'http://www.gooseplanet.ru/'
});

TypeError: _gooseParser.PhantomEnvironment is not a constructor

I look at the imported entities and both of them is undefined.

If I write:
import Parser from 'goose-parser'

It return [Function: Parser]
But where I can find PhantomEnvironment???

Add ability to execute custom actions before start parsing.
For example we need to

• land on search page (provided URL)
• insert in the field search text
• press to search button
• land to search results page and start parsing there

This functionality will completely replace actions with once flag

Add ability to paginate via ajax pages

Move current test system to new efficient way (Take a look on new tests here #76)

So, we have:
old tests here: tests/phantom_parser_test.js
new tests here: tests/phantom/lib/

To run tests, just call npm test

When move some test, remove it in tests/phantom_parser_test.js
At the end remove a file tests/phantom_parser_test.js and html page for it.

Add to docs:

• Describe Pagination.
• Describe Parser.