website-scraper / demo Goto Github PK

502 Bad Gateway

What file do I change to enable the iframe links to work?

Hi, I'm using the library to scrape a site and I need all the assets on their root folders:

css/[all CSS here]
js/[all JS here]

Turns out that it works, but in some cases it creates subdirectories with the very same structure. I want the scrapper to put all configured assets (no matter the deep page it belongs to) to be on the root folder.

Some websites will determine the country of the IP address.

where is the variable in the javascript that stores the html of the fetched scraped page? need to find it so I can modify the content.

I think, dependencies update are required.

Also, I get bower question, when I trying install app:

Unable to find a suitable version for angular, please choose one:
    1) angular#1.2.29 which resolved to 1.2.29 and is required by angular-cookies#1.2.29, angular-route#1.2.29
    2) angular#~1.3.15 which resolved to 1.3.20 and is required by web-scraper
    3) angular#1.3.20 which resolved to 1.3.20 and is required by angular-resource#1.3.20

Configuration

version: [result of npm ls website-scraper --depth 0 command]

options: [provide your full options object]

i tried your demo app and the problem is that there is only  the main  url which is not downloaded under url for example if the url is a.html it will not download a / 2.html

Description

[Description of the issue]

Expected behavior: [What you expect to happen]

Actual behavior: [What actually happens]

Additional Information

[Any additional information, configuration or data that might be necessary to reproduce the issue]

Really weird occurance. Scraper ran, clicked on the new window button to preview, everything looks normal.

Download the file, the background images have disappeared..

Another example:

https://scraper.nepochataya.pp.ua/static/files/www.riemtex.de-1594223643199/

After download:

Demo is gone, https://scraper.nepochataya.pp.ua/

I'm learning your script and looking at the demo app. And noticed that some sites are being opened in full screen not iframe for example this page "http://www.delfi.lt/", which is a news page.
I'm not sure but the problem might be here

Can't Load URL: The domain of this URL isn't included in the app's domains. ping?client_id=161041023936278&domain=localhost&origin=1&redirect_uri=http%3A%2F%2Fstaticxx.faceboo…:1 To be able to load this URL, add all domains and sub-domains of your app to the App Domains field in your app settings.

OneSignal: Could not load iFrame with URL https://delfi.onesignal.com/webPushIframe. Please check that your 'subdomainName' matches that on your OneSignal Chrome platform settings. Also please check that your Site URL on your Chrome platform settings is a valid reachable URL pointing to your site.

When I open scraped page, like this, and click Back button I should be returned on list page, not main.