Code Monkey home page Code Monkey logo

usus's Introduction

ūsus

Travis build status Coveralls NPM version Canonical Code Style Twitter Follow

Webpage pre-rendering service. ⚡️

Deprecated

Deprecated in favour of https://github.com/GoogleChrome/rendertron

The two projects are not functionally interchangeable. However, the overlap is significant enough to discourage the duplication of effort.

Future releases of ūsus will be limited to bug fixes.

Features

  • Renders webpage using the Chrome Debugging Protocol (CDP).
  • Extracts CSS used to render the page.
  • Renders HTML with the blocking CSS made asynchronous.
  • Inlines the critical CSS.
  • Preloads CSS and fonts used to render the page using rel=preload.

Motivation

Static HTML pages with inline CSS load faster and are better indexed than single page applications (SPA). ūsus pre-renders single page applications into static HTML with the critical CSS inlined.

Removing the blocking CSS and inlining the CSS required to render the page increases the perceived page loading speed. Presumably, improves SEO by reducing the page loading time.

Read Pre-rendering SPA for SEO and improved perceived page loading speed.

Demo

Examples of web pages using ūsus:

Use cases

  • Produce HTML used to render the page. Used to render single page applications (e.g. React and Angular) to produce a static HTML. This can be used as a replacement of https://prerender.io/. Default behaviour.
  • Extract CSS used to render a specific page. Used to capture the critical CSS. Use --extractStyles option.
  • Produce HTML used to render the page with the critical-path CSS inlined and blocking CSS made asynchronous. Use --inlineStyles option.

Node.js API

ūsus can be used either as a Node.js dependency or as a CLI program.

import {
  render
} from 'usus';

/**
 * @see https://github.com/gajus/usus#configuration
 */
const configuration: UserConfigurationType = {}

const css = await render('http://gajus.com/', configuration);

Configuration

// Flow type annotations included for user reference only.
// ūsus does not depend or require use of Flow type.
//
// Refer to the table below for an alternative form of documentation.

type CookieType = {|
  +name: string,
  +value: string
|};

export type UserDeviceMetricsOverrideType = {
  +deviceScaleFactor?: number,
  +fitWindow?: boolean,
  +height?: number,
  +mobile?: boolean,
  +width?: number
};

type FormatStylesType = (styles: string) => Promise<string>;

export type UserConfigurationType = {
  +chromePort?: number,
  +cookies?: $ReadOnlyArray<CookieType>,
  +delay?: number,
  +deviceMetricsOverride?: UserDeviceMetricsOverrideType,
  +extractStyles?: boolean,
  +formatStyles?: FormatStylesType,
  +inlineStyles?: boolean,
  +preloadFonts?: boolean,
  +preloadStyles?: boolean
};

The default behaviour is to return the HTML.

  • Using the --extractStyles option returns the CSS used to render the document.
  • Using the --inlineStyles option returns HTML document with CSS inlined.
Name Type Description Default value
chromePort number Port of an existing Chrome instance. See Controlling the Chrome instance. N/A
cookies Array<{name: string, value: string}> Sets a cookie with the given cookie data. N/A
delay number Defines how many milliseconds to wait after the "load" event has been fired before capturing the styles used to load the page. This is important if resources appearing on the page are being loaded asynchronously. number
deviceMetricsOverride See deviceMetricsOverride configuration
extractStyles boolean Extracts CSS used to render the page. false
formatStyles (styles: string) => Promise<string> Used to format CSS. Useful with inlineStyles=true option to format the CSS before it is inlined. N/A
inlineStyles boolean Inlines the styles required to render the document. false
preloadFonts boolean Adds rel=preload for all fonts required to render the page. true
preloadStyles boolean Adds rel=preload for all styles removed from <head>. Used with inlineStyles=true. true
url string The URL to render. N/A

deviceMetricsOverride configuration

Name Type Description Default value
deviceScaleFactor number Overriding device scale factor value. 1
fitWindow boolean Whether a view that exceeds the available browser window area should be scaled down to fit. false
height number Overriding width value in pixels (minimum 0, maximum 10000000). 1080
width number Overriding height value in pixels (minimum 0, maximum 10000000). 1920
mobile boolean Whether to emulate mobile device. This includes viewport meta tag, overlay scrollbars, text autosizing and more. false

For more information about the deviceMetricsOverride configuration, refer to Chrome DevTools Protocol Viewer documentation.

Installation

Using ūsus requires to install usus NPM package and Google Chrome browser (refer to Dependencies).

Dependencies

ūsus depends on Chrome v59+.

For Docker installation instructions, refer to Building Docker container with Chrome.

Cookbook

Using via the command line interface (CLI)

$ npm install usus --global
$ usus --help
# usus <command> --help
$ usus render --help
# Renders static HTML. Equivalent to https://prerender.io/.
$ usus render --url http://gajus.com/
# Inlines styles required to render the page.
$ usus render --url http://gajus.com/ --inlineStyles true
# Use cookies when loading the page.
$ usus render --url http://gajus.com/ --cookies foo=bar,baz=qux
# Render emulating a mobile device (example is using iPhone 6 parameters).
$ usus render --url http://gajus.com/ --deviceMetricsOverride.deviceScaleFactor 2 --deviceMetricsOverride.fitWindow false --deviceMetricsOverride.height 1334 --deviceMetricsOverride.mobile true --deviceMetricsOverride.width 750

Building Docker container with Chrome

Add the following line to your Dockerfile:

RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
  && sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
  && apt-get update -y \
  && apt-get install google-chrome-stable -y

This assumes that you are extending from the base node image.

Controlling the Chrome instance

By default, ūsus creates a new instance of Chrome for every render operation and destroys it after completion. However, you can start Chrome independent of ūsus and re-use the same instance for multiple renderings.

import {
  launchChrome,
  render
} from 'usus';

const chrome = await launchChrome();

await render('https://go2cinema.com/movies/baywatch-2017-1198354', {
  chromePort: chrome.port,
  inlineStyles: true
});

await render('https://go2cinema.com/movies/baby-driver-2017-2257838', {
  chromePort: chrome.port,
  inlineStyles: true
});

await chrome.kill();

launchChrome is a convenience method to launch Chrome using default ūsus configuration. If you need granular control over how Chrome is launched, refer to the chrome-launcher program.

Minifying the CSS

Use the formatStyles callback to minify/ format/ optimize/ remove CSS before it is inlined.

In this example, I am using csso minifier.

import {
  render
} from 'usus';
import {
  minify
} from 'csso';

await render(url, {
  formatStyles: (styles: string): Promise<string> => {
    return minify(styles).css;
  },
  inlineStyles: true
});

Debugging

Export DEBUG=usus variable to get additional debugging information, e.g.

$ export DEBUG=usus*
$ usus --url http://gajus.com

Implementation

ūsus uses Chrome Debugging Protocol CSS coverage report to generate a stylesheet used to render the document.

Alternatives

The following programs provide equivalent service:

All of these programs are using PhantomJS or JSDom to render the page/ evaluate the scripts.

ūsus is different because it is using Chrome Debugging Protocol and it leverages the Chrome CSS coverage report to generate a stylesheet used to render the document.

usus's People

Contributors

davidosomething avatar gajus avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

usus's Issues

What makes styles 'critical'?

I'm not sure how usus (CSS Coverage, really) classifies styles as 'critical'.
Most other tools, I believe, look at the styles needed to render the first screen (above the fold) of content.

usus seems to take more styles as critical?

CentOS - ECONNREFUSED ip:port

When I use the utility in CentOS it never works. The solution was found here

Error: connect ECONNREFUSED 127.0.0.1:41730
    at Object._errnoException (util.js:1024:11)
    at _exceptionWithHostPort (util.js:1046:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1182:14)


There's need in '--no-sanbox' launch option. I have not investigated the reasons so I'd like just to offer enable the option by default.

Multiple viewports, mobile + non-mobile

Is there a way to call usus to obtain a single stylesheet with "critical" CSS for all possible viewport sizes, and mobile and retina options?

Motivation: We don't know the client's viewport size on the server side, so the critical CSS should include the necessary CSS for all viewport sizes.

Add tests

Need to write tests for Chrome rendering.

[Ask] How to Open Raw HTML (.html) in React 'Public' Folder?

I've posted this question to Stackoverflow but still no luck :
Maybe this is not directly related to "usus", but have no luck Google for the answer.

Question : How can I access .html file which have been created by render() function of usus in create-react-app public folder /public/

I'm wondering how @gajus make this >> https://go2cinema.com/movies/baywatch-2017-1198354

Long Question here : https://stackoverflow.com/questions/48426119/how-to-open-raw-html-html-in-react-public-folder

preload non-critical CSS pattern?

Using inlineStyles I noticed you inline the 'critical' CSS in a <style> block in the <head> and then <link> the complete, non-critical, CSS file right above </body> (so as not to block rendering).

Another good pattern is to use rel=preload in the head for this non-critical CSS file and then load it with JS. A good explanation (and JS utility) can be found in LoadCSS by the Filamentgroup.

Adding a loadCSS script might be out of scope, but it might be good to document (README?) this usage with extractStyles.

inlineStyles could actually do half the work by also adding a <link rel="preload" href="org-stylesheet.css" as="style">.

Basic example not working

 alessio: /home/alessio/www/tmp 
→ usus render --url http://gajus.com/
  ChromeLauncher Waiting for browser. +0ms
  ChromeLauncher Waiting for browser... +1ms
  ChromeLauncher Waiting for browser..... +521ms
  ChromeLauncher Waiting for browser.....✓ +1ms
/home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/dist/bin/index.js:15
  throw error;
  ^

Error: 'CSS.takeCoverageDelta' wasn't found
    at /home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/node_modules/chrome-remote-interface/lib/chrome.js:90:30
    at Chrome.handleMessage (/home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/node_modules/chrome-remote-interface/lib/chrome.js:289:13)
    at WebSocket.<anonymous> (/home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/node_modules/chrome-remote-interface/lib/chrome.js:266:27)
    at emitTwo (events.js:125:13)
    at WebSocket.emit (events.js:213:7)
    at Receiver._receiver.onmessage (/home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/node_modules/ws/lib/WebSocket.js:143:54)
    at Receiver.dataMessage (/home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/node_modules/ws/lib/Receiver.js:385:14)
    at extension.decompress (/home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/node_modules/ws/lib/Receiver.js:354:40)
    at _inflate.flush (/home/alessio/.nvm/versions/node/v8.1.3/lib/node_modules/usus/node_modules/ws/lib/PerMessageDeflate.js:279:12)
    at afterWrite (_stream_writable.js:438:3)
    at onwrite (_stream_writable.js:429:7)
    at afterTransform (_stream_transform.js:102:3)
    at TransformState.afterTransform (_stream_transform.js:75:12)
    at Zlib.callback (zlib.js:485:5)

 alessio: /home/alessio/www/tmp 
→ node -v && npm -v
v8.1.3
5.0.3

media queries not included

hey,

your project inspired me, but my usecase is a bit out of scope, so I created my own package...

However, I found out, that CSS.takeCoverageDelta() doesn't include the media queries for a rule.
I tinkered a bit and outsourced the resulting parser in a separate package coverage-delta-parser. Feel free to use or copy, it is rather small..

Create a logo

Need a logo for ūsus. Something simple; might refer to Chrome and CSS.

Custom page load event.

Hello. Is there any way to provide some custom event that will indicate that page is loaded? For example, by setting some variable or firing custom event?

As an example you may refer to chrome-render lib (https://github.com/gwuhaolin/chrome-render):

useReady: boolean whether use window.isPageReady=1 to notify chrome-render page has ready. default is false chrome-render use domContentEventFired as page has ready.

Unexpected token error

c:\Temp\render>usus render --url http://myreactsite.com
C:\Users\me\AppData\Roaming\npm\node_modules\usus\node_modules\bluefeather\dist\map.js:9
const map = async (values, mapper, configuration) => {
                  ^
SyntaxError: Unexpected token (
    at createScript (vm.js:56:10)
. . .

node --version
v6.10.2

Probably just need to define the version of node required (v7+) in the dependencies

Style does not get inlined

for the site https://yarnpkg.com/en/package/vue-instantsearch, inlineStyles and extractStyles don't have an effect on the page:

vue-instantsearch master ✗ ➜ usus render --url https://yarnpkg.com/en/package/vue-instantsearch > vue-instantsearch.html
  ChromeLauncher Waiting for browser. +0ms
  ChromeLauncher Waiting for browser... +1ms
  ChromeLauncher Waiting for browser..... +509ms
  ChromeLauncher Waiting for browser.....✓ +3ms
  ChromeLauncher Killing all Chrome Instances +7s
vue-instantsearch master ✗ ➜ code .
vue-instantsearch master ✗ ➜ usus render --url https://yarnpkg.com/en/package/vue-instantsearch --inlineStyles true> vue-instantsearch.html
  ChromeLauncher Waiting for browser. +0ms
  ChromeLauncher Waiting for browser... +1ms
  ChromeLauncher Waiting for browser..... +512ms
  ChromeLauncher Waiting for browser.....✓ +2ms
  ChromeLauncher Killing all Chrome Instances +9s
vue-instantsearch master ✗ ➜ usus render --url https://yarnpkg.com/en/package/vue-instantsearch --extractStyles true> vue-instantsearch.html
  ChromeLauncher Waiting for browser. +0ms
  ChromeLauncher Waiting for browser... +1ms
  ChromeLauncher Waiting for browser..... +507ms
  ChromeLauncher Waiting for browser.....✓ +2ms
  ChromeLauncher Killing all Chrome Instances +8s
vue-instantsearch master ✗ ➜ usus render --url https://yarnpkg.com/en/package/vue-instantsearch --extractStyles=true> ~/Desktop/vue-instantsearch.html
  ChromeLauncher Waiting for browser. +0ms
  ChromeLauncher Waiting for browser... +1ms
  ChromeLauncher Waiting for browser..... +513ms
  ChromeLauncher Waiting for browser.....✓ +2ms
  ChromeLauncher Killing all Chrome Instances +12s
vue-instantsearch master ✗ ➜ usus render --url https://yarnpkg.com/en/package/vue-instantsearch --inlineStyles=true> ~/Desktop/vue-instantsearch.html
  ChromeLauncher Waiting for browser. +0ms
  ChromeLauncher Waiting for browser... +1ms
  ChromeLauncher Waiting for browser..... +510ms
  ChromeLauncher Waiting for browser.....✓ +3ms
  ChromeLauncher Killing all Chrome Instances +11s

Gist of the output here and live on rawgit

Engines definition excludes node 6

I may be mistaken, but this module seems to work fine with node 6? However, when I try to install it with yarn, I get the following error:

error [email protected]: The engine “node” is incompatible with this module. Expected version “>6".

The engines definition in my project is the following:

  "engines": {
    "node": "^6.10.3"
  },

Is this just a matter of changing the package.json in this repo to allow 6? It would be nice to not force all our engineers to use --ignore-engines!

Skip `inlineStyles` when already inline?

This could be seen as a user error, but I noticed that inlineStyles basically duplicated my already-inline styles.

For my site I have all styles as an inline style block in the head. So no external <link>.

usus render --url https://valuedstandards.com/ --inlineStyles true > valuedstandards.html

.. will result in duplicate <style> blocks in the <head> :-)

Again: this might be to be expected since I should not have used inlineStyles, but it would be neat if this could be caught.

Prevent inline styling

Hi!

when I run usus without any flag the output has all my styles inlined, which is undesirable as my html files grows up to 5000 LOC.

Shouldn't inline styling be disabled by default? Am I doing something wrong? I wrapped usus in a webpack plugin, but I'm just running it like this, nothing fancy

const Usus = require('usus')
Usus.render('someUrl')

thanks for your time!

Create examples for Node JS SEO crawlers use

While this works great with the terminal example (it returns the desired HTML), I'm not sure how this can be implemented on, say a basic express app, where I want the crawlers to get the prerendered HTML for SEO purposes. One or two examples as to how to implement this would be awesome!

Other than that the project seems great!

Pseudo classes missing from extracted CSS?

I noticed extractStyles and inlineStyles does not produce any pseudo classes. E.g. a:hover was left out.

I realize this is all not usus specific (but CSS Coverage) but I was wondering why. Also: for critical styles I would expect these to be included.

Does not work with Node v6

.. which is the LTS (still recommended for most users):

→ usus --url http://gajus.com
/home/alessio/.nvm/versions/node/v6.9.4/lib/node_modules/usus/dist/bin/commands/render.js:72
const handler = exports.handler = async argv => {
                                        ^^^^
SyntaxError: Unexpected identifier
    at Object.exports.runInThisContext (vm.js:76:16)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at Object.require (internal/module.js:20:19)
    at /home/alessio/.nvm/versions/node/v6.9.4/lib/node_modules/usus/node_modules/require-directory/index.js:76:17
    at Array.forEach (native)

Postprocessing url() in result critical css

Hello, thanks for the work you do: it's great!
But I have an issue I hope you help with:
Is there a convenient way to rewrite relative paths in url()?
The critical css is placed in head section and relative paths copied from css files are usually broken due to the files are located in different path than index.php or something?

Provide a lower-level API/ Bring your own Chrome instance

A lower-level API should allow to control when to start Chrome and kill it, as opposed to restarting the app on every request. This would allow more scalable use cases.

The lower-level API should expose a direct access to Chrome components.

Add an option to preload fonts using rel=preload

ūsus can parse the CSS, find the fonts used, and preload the fonts using rel=preload.

<link rel="preload" href="/assets/myfont.woff" as="font">

There can be two options:

  • remove font loading from extracted CSS altogether (because some don't consider it part of the critical path);
  • remove font loading from the extracted CSS and add rel=preload to HTML

Idea thanks to #20 and #8 discussions.

Detect if a page responded with a 404 statusCode

Hi Gajus, I cannot thank you enough for this project.
I'm trying to detect if a crawled page responds with a statusCode different than 200 for SEO reasons, is that possible / will be possible?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.