Comments (20)
@geshan Have you been emailing with @KittyBot? This looks great, next steps are:
- Transfer the repository to me (mrkurt)
- We'll work out payment over email.
from fly.
Good deal! It's here now, I'll get it in our docs soon: https://github.com/fly-examples/puppeteer-js-renderer
from fly.
This sounds super cool. It's basically a service that executes JS and then returns the resulting DOM? I think with a README about why that's interesting and how it's better if you run it close to certain cities, that's a pretty great example.
I've actually wanted examples of Puppeteer for other stuff:
- Screenshots/thumbnails
- Lighthouse tests
Could be good for a second example. ;)
from fly.
Hey @mrkurt , appreciate your fast reply.
It's basically a service that executes JS and then returns the resulting DOM?
- Yes you are right :)
Here is the repo with steps on how to get this app running on fly.io - https://github.com/geshan/js-renderer-fly . Let me know what would be the next step(s) to get it on fly-examples. I am open to editing the Readme too.
I can do a screenshot as a service example as the next one. Thanks!
from fly.
Hi, this is a good start - I have some notes for you -
The opener is talking to an audience that already knows what the problem is, and even what the most common tool
Suggest that it might start up something like
"JavaScript is the bane of a web scrapers life. Scraping is all about extracting data from a web page and JavaScript is there adding content, hiding blocks, moving the DOM around and just reading the HTML from the server is just not enough. What you ideally want is a way to run all that JavaScript on the page so you can see what's left after that. Then you can get down to some serious scraping.
There's tools to do this out there but most have their own compliactions or restrictions that stop them from being used out on the edge. Js-renderer-fly has none of those problems and with Fly, you can deploy to close to your users too."
(Roll in the Uses section here, with a practical example - maybe scrape Instagram data and product a list of pics)
How to deploy it on Fly - move 1 and 2 into prerequisits...
3 - only works if you are logged in the SSH support enabled
5/6 - Run flyctl init
- hit return for a app name to be generated (unless there's a name you really want)
You can add flyctl init —dockerfile to skip the picking of the builer
Also, re orgs - first one on the list will be your own org now
Not sure the deploy screenshot adds much - maybe explain the stages briefly? The details aren't added to the fly.toml file, they come from the fly.toml file
11 Not sure what you are saying there.
A tour of commands might be good at this point.... status, restart, pause? Leading into the scale commands and a regions command to put an instance on every continent
A script to do something fun with IG or similar to wrap up completing the task from the start?
from fly.
@codepope I have made the suggested changes here: https://github.com/geshan/js-renderer-fly/pull/6/files let me know if it is ok, thanks!
from fly.
Make a branch and merge the changes into that branch. It's difficult the review article/readme content with just patch files.
Some quick notes though. Explain what puppeteer is, or at least link to it. (see previous note on audience).
from fly.
@codepope merged to master, it can be see here: https://github.com/geshan/js-renderer-fly . I will add a bit ore details about puppeteer soon. If anything else needs to be added, please let me know, thanks!
from fly.
"Scraping is all about extracting data from a web page and JavaScript is there adding content, hiding blocks, moving the DOM around and just reading the HTML from the server is just not enough."
change to
"Scraping is all about extracting data from a web page and JavaScript is there adding content, hiding blocks and moving the DOM around. Just reading the HTML from the server is just not enough."
"There are tools to do this out there but most have their own compliacations or restrictions that stop them from being used out on the edge. Js-renderer-fly has none of those problems and with Fly, you can deploy to close to your users too.
This is an online puppeteer service to render pages with javascript (js) very useful for web scraping." - pull together to make one para... something like
"There are tools to do this out there but most have their own complications or restrictions that stop them from being used out on the edge. Js-renderer-fly has none of those problems and with Fly, you can deploy to close to your users too. At its core, js-renderer-fly is a puppeteer-based service. Puppeteer is a package which renders pages using a headless Chrome instance, executing the JavaScript within the page."
Uses section seems redundant. Maybe blend with the Quick Try....
Explain that a typical Youtube page adds the view count in JavaScript and to get that value, we're going to use js-renderer-fly to pull out that value after the JavaScript has run.
The quick-try ideally should prompt the user at that point to clone the github repo.
Will have to go over it for typos and things like "Then select and org"....
Pull the resources section into the "More Fly Commands" section so you cover lifecycle, vertical scaling and global scaling.
That's about it for now.
from fly.
Hi @codepope , I have done most of the changes: https://github.com/geshan/js-renderer-fly.
I have done a quick typo and grammar fix with grammarly, thanks for the ping.
The quick-try ideally should prompt the user at that point to clone the github repo.
this is the part I am not clear about. So this node script should clone this repo and try to deploy it for the user?
Let me know if more changes are required, thanks!
from fly.
At that point, reading through, the user will not have downloaded anything. So, you'd likely want to suggest they either grab the script from the repo or clone the repo before discussing the script.
from fly.
@codepope Just fixed that part too, thanks for the ping it made sense. Latest changes are here: https://github.com/geshan/js-renderer-fly . Open to suggestions.
from fly.
@codepope feedback is welcome :)
from fly.
generally youtube -> YouTube
The installer line
git clone [email protected]:geshan/js-renderer-fly.git && cd js-renderer-fly && npm install && node yt-views.js
is a bit daunting and probably better broken down into separate lines
Also the scraper app is a bit of a black box - maybe a line or two about what it does? (and a mention for the axrios library) so people could use it to kickstart actually writing a scraper.
Run Locally now repeats those instructions too...
Suggest make "Quick Try" a "Quick Start" and add a sub section for installing, and a sub section for "Your first scraping" or something. "Use it as a service" should point out that the instructions later will show you how to deploy it as a service and that you are just showing how it works with an already deployed version.
"if you are logged in the SSH support enabled else try" else->otherwise
" I tried with: " -> "I ran it with js-renderer-fly as the app name for the examples"
"Subsequently, you can select an organization. Generally, it will be your first name-last name on the prompt" Generally, it->Usually, this will...
Step 9 is confusing. I think you are trying to get two things over at once. Also it's not clear what the command line should be flyctl open /api/render?url=<your-url>
I assume
You may want to give a reason why you would want to suspend the service
"So I wanted to check how much resources were allocated to this app on fly by default. It was easy to know with the following commands" -> "I wanted to see what resources were allocated to the App on Fly. The scale
commands allowed me to find out"
"Now your service is running well in one data center for me it was iad which is Ashburn, Virginia (US). Now let's add some more:"
Each sentence starts with Now. "Our service is now running in one data center. For me, it's iad (Ashburn, Virginia) but yours will likely be different based on where you are working from. We can add instances around the world the speed up responses...
from fly.
@codepope appreciate you taking the time for such detailed feedback, all of it has been updated: https://github.com/geshan/js-renderer-fly . Let me know if it needs any more improvements, thanks!
from fly.
Just a ping for this @codepope :).
from fly.
Thanks for the PRs @codepope , both have been merged. What would the next step be?
from fly.
@mrkurt Yes I am in comms with @KittyBot . I have transferred the repo to you, thanks a lot!
from fly.
Alrighty, now in the docs. I renamed the project and did a little cleanup of project names in the README text, but check it over and see if I missed anything: http://fly.io/docs/app-guides/puppeteer-js-renderer/
Feel free to submit PRs to the project for any changes you want to make.
from fly.
Looks really nice. There were some references to the old project name and Github URL, I have changed them in this PR, thanks!
from fly.
Related Issues (20)
- Running Varnish on Fly HOT 2
- Secured API Key generation on the edge with Fly HOT 2
- Continuous deployment + [Various] HOT 2
- Continuous deployment with CircleCI HOT 8
- Continuous Deployment with Travis HOT 13
- Continuous Deployment using Github Actions HOT 1
- Continuous Deployment with Codeship HOT 1
- Continuous Deployment with Buddy.works HOT 1
- Using OpenResty for Authentication on Fly HOT 6
- Text summarizer (based on BERT) as a service/API on fly HOT 10
- Data[base|store]s & Fly HOT 11
- Redis cache cluster HOT 3
- Web app firewall HOT 1
- Rust on Fly
- GraphQL at Edge HOT 1
- hands-on docs bug
- allow exposing multiple port range in machines API and fly.toml
- Machine API V2 and region add does not scale as expected HOT 1
- Typo on `App status: Suspended` description HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fly.