Code Monkey home page Code Monkey logo

virtual-scroller's Introduction

A built-in virtual scroller for the web platform

This repository hosts explorations for a new web platform feature, a virtual scroller control. The idea of a virtual scroller is to provide a scrolling "viewport" onto some content, allow extremely large numbers of elements to exist, but maintain high performance by only paying the cost for those that are currently visible. Traditionally, we say that the non-visible content is virtualized.

Current status

This project is no longer being actively developed. Instead we are pursuing primitives here.

Why a virtual scroller?

Virtualized content is a popular and important pattern on the web. Most content uses it in some form: the https://m.twitter.com and https://facebook.com feeds; Google Photos and YouTube comments; and many news sites which have an automatic "scroll to next article" feature. Most popular frameworks have at least one high-usage virtualization component, e.g. React Virtualized with ~290K weekly downloads. See the infinite list study group for more research on existing virtualization solutions, both on the web and in other platforms like iOS.

At the same time, virtual scrollers are complex and hard to get right. In fact, having a first-class experience with virtualized content is currently impossible, because browsers don't expose the right hooks: things like accessible landmark navigation, find in page, or intra-page anchor navigation are based solely on DOM structure, and virtualized content is by definition not in the DOM. Additionally, today's virtualized content does not work with search engine crawlers, which means that sites that care about search engine ranking are unable to apply this important performance technique. This is bad for the web.

We believe that, like native platforms, the web deserves a first-class virtual scroller implementation that works out of the box.

For more details on the motivation, see Motivation.md.

Sample code

<!--
  virtual-scroller lives in a built-in module and needs to be imported before use.
  (The name of the module is subject to change.)
-->
<script type="module">
import "std:virtual-scroller";
</script>

<!--
  The <virtual-scroller> will manage the rendering of its children.
  It will prioritize rendering things that are in the viewport and not render
  children that are far away, such that we are only paying as little rendering
  cost as possible while still allowing them to work with find-in-page,
  accessibility features, focus navigation, fragment URL navigation, etc.
-->
<virtual-scroller id='scroller'>
  <div>Item 1</div>
  <div>Item 2</div>
  ...
  <div>Item 1000</div>
</virtual-scroller>

<script>
// You can add, remove, modify children of the <virtual-scroller> as you would
// a regular element, using DOM APIs.
scroller.append(...newChildren);

// When the set of actually-rendered children is about to change,
// the <virtual-scroller> will fire a "rangechange" event with the
// new range of rendered children.
scroller.addEventListener('rangechange', (event) => {
  if (event.first === 0) {
    console.log('rendered first item.');
  }
  if (event.last === scroller.children.length - 1) {
    console.log('rendered last item.');
    // Perhaps you would want to load more data for display!
  }
});
</script>

Goals

  • Be flexible enough to be used for various kinds of scrolling, larger-than-viewport content, from news articles to typeaheads to contact lists to photo galleries.
  • Only pay the rendering costs for visible child elements of the scroller (with some margin).
  • Allow contents of the scroller to work with find-in-page, accessibility features, focus, fragment URL navigation, etc., just as they would in a non-virtualized context.
  • Be flexible enough to allow developers to integrate advanced features and behaviors, such as groupings, sticky headers, animations, swiping, item selection, etc.
  • Support 1D horizontal/1D vertical/2D wrapped-grid layouts.

Non-goals

  • Allow data that are not part of the DOM to work with find-in-page, accessibility, etc. The DOM remains the source of truth for these browser features, not e.g. the server-side, or other in-memory JavaScript data structures. See below for more elaboration.

And, at least in v1, the following are out of scope:

  • Built-in support for advanced features and behaviors, such as those mentioned above.
  • Support infinite-grid (spreadsheet-like), multi-column (content continues from bottom of col(N) to top of col(N+1)) or masonry layouts.

Proposed APIs

<virtual-scroller> element

The <virtual-scroller> element represents a container that will manage the rendering of its children. The children of this element might not get rendered/updated if they are not near or in the viewport. The element is aware of changes to the viewport, as well as to its own size, and will manage the rendered state of its children accordingly.

Its children can be any element. Semantically, the element is similar to a <div>, with the addition of focus on only the visible contents when the element overflows its container and is scrolled.

The sizes and other layout values of the non-rendered children, which might affect scrollbar height, etc., are approximate sizes and might not be always accurate. The rendered children always have accurate style and layout values, just like other normal DOM nodes.

All children, rendered or non-rendered, will work with find-in-page, focus navigation, fragment URL navigation, and accessibility technology, just like normal DOM nodes.

rangechange event

Fired when <virtual-scroller> is about to render a new range of items, e.g. because the user scrolled. This will fire at requestAnimationFrame timing, i.e. before the browser is about to paint.

The event has the following properties:

  • first: an integer, the 0-based index of the first children currently rendered.
  • last: an integer, the 0-based index of the last children currently rendered.
  • bubbles: false
  • cancelable: false
  • composed: false

As an example, this can be used to delay more costly rendering work. For example, a scrolling code listing could perform just-in-time syntax highlighting on lines right before they become visible, leaving the un-adorned code accessible by find-in-page/etc. but improving the code's appearance before the user sees it.

TODO: do we need first and last on the event, or should we just use the properties of the element?

rangeFirst and rangeLast getters

These return 0-based indices giving the first and last children currently rendered.

TODO: these names are kind of bad?

Constraints and effects

Ideally, we would like there to be zero constraints on the contents of the <virtual-scroller> element, or on the virtual-scroller element itself.

Similarly, we would like to avoid any observable effects on the element or its children. Just like how <select> does not cause its <option> elements to change observably when you open the select box, ideally <virtual-scroller> should not cause observable effects on its children as the user scrolls around.

This may prove difficult to specify or implement. In reality, we expect to have to add constraints such as:

  • Overriding the default values for certain CSS properties (and ignoring web developer attempts to set them).
  • Having degenerate behavior if visual order does not match DOM order (e.g. via flex-order or position: absolute).

And the control may influence its children via effects such as:

  • Changing the computed style of children (observable via getComputedStyle(child)).
  • Changing the display-locked status of children (observable via child.displayLock.locked).
  • Changing the layout of non-visible children's descendants (observable via e.g. child.children[0].getBoundingClientRect()).

Figuring out the exact set of constraints (including what happens when they're violated), and the exact set of effects, is a key blocker for standardization that we expect to address over time.

Use cases

Cases this proposal covers well

This design is intended to cover the following cases:

  • Short (10-100 item) scrollers. Previously, virtualizing such scrollers was done rarely, as virtualization caused sacrifices in developer and user experience. We are hopeful that with a first-class virtualization element in the web platform, it will become more expected to use <virtual-scroller> in places where overflow-scrolling <div>s were previously seen, thus improving overall UI performance.

  • Medium (100-10 000 item) scrollers. This is where virtual scrollers have traditionally thrived. We also want to expand this category to include not just traditional list- or feed-like scenarios, but also cases like news articles.

  • Large (10 000+ item) scrollers, where data is added progressively. As long as the data can be held in memory, <virtual-scroller> ensures that there are no rendering costs, and so can scale indefinitely. An example here would be any interface where scrolling down loads more content from the server, indefinitely, such as a social feed or a busy person's inbox.

    However, note that adding a large amount of data at once is tricky with this API; see below.

Very large amounts of initial data

Consider the cases of the singlepage HTML specification, or of long-but-finite lists such as a large company directory.

We believe that all these scenarios are still suited for use with this virtual scroller control. As long as the data could feasibly fit in memory in any form, the user experience will be best if it is stored in the DOM, inside a virtual scroller that makes the rendering costs of out-of-viewport items zero. This allows access to the data by browser features, such as find-in-page or accessibility tooling, as well as by search engines.

However, using the above API in these scenarios suffers from the problem of initial page load costs. Trying to server-render all of the items as <virtual-scroller> children, or trying to do an initial JSON-to-HTML client-render pass, will jank the page. For example, just the parsing time alone for the single-page HTML specification can take 0.6–4.4 seconds. And there are staging problems in trying to deliver large amounts of HTML while the "std:virtual-scroller" module is still being imported, which could prevent it from properly avoiding initial rendering costs.

As such we think there is still room for improvement in these scenarios, e.g. with an API that makes it easy to progressively stream data during idle time to allow the initial few screenfuls to render ASAP and without jank. We will be exploring this problem over time, after we feel confident that we can specify and implement a solution for the core use cases.

Almost-infinite data from the server

A consistent point of confusion about the virtual scroller proposal is how it purports to solve cases like social feeds, where there is an "almost infinite" amount of data available.

This proposal's answer is that: if you were going to have the data in memory anyway, then it should be in the <virtual-scroller>, and thus accessible to the browser or other technologies (such as search engines) that operate on the DOM. But, if you were going to leave the data on the server, then it is fine to continue leaving it on the server, even with a <virtual-scroller> in play.

For example, in one session while browsing https://m.twitter.com/, it limited itself to only keeping 5 tweets in the DOM at one time, using traditional virtualization techniques. However, it appeared to have about 100 tweets in memory (available for display even if the user goes offline). And, when the user began scrolling toward the bottom of the page, it queried the server to increase the amount of in-memory tweets it had available. With a native <virtual-scroller> in the browser, which mitigates the rendering costs while still allowing you to keep items in the DOM, we're hopeful that it'd be possible to keep those 100+ tweets as DOM nodes, not just in-memory JavaScript values that are locked away from find-in-page and friends.

This proposed design does mean that there could be things on the Twitter servers which are not findable by find-in-page, because they have not yet been pulled from the server and into the DOM. That is OK. Find-in-page is not meant to be find-in-site, and users of social feeds are able to understand the idea that not everything is yet loaded. What is harder for them to understand is when they saw a phrase, they scroll past it by 100 pixels, and then find-in-page can't see it anymore, because it's been moved out of the DOM. <virtual-scroller> addresses this latter problem.

Alternatives considered

Using traditional virtualization

Previously, we intended to specify a traditional approach to virtualization for the built-in virtual scroller. With that approach, the element would map JavaScript values ("items") to DOM element children, putting only a small portion of the items in the DOM, with callbacks for creating, updating, and recycling the DOM elements given an item.

However, this approach suffers the same problem as existing traditionally-virtualized scrollers regarding accessibility, find-in-page, fragment URL and focus navigation, etc., all of which depend on having the content be part of the DOM to work correctly. This is a known issue with traditional virtualization, which web developers have to grapple with today, trading off these functionalities with the performance improvement. As we intend for the built-in virtual scroller to be a standard building block that a lot of web authors would use or build on, we don't want to continue having this disadvantage.

In other words, given the problem of too much DOM causing bad performance, traditional virtualization is managing the symptoms, by decreasing the amount of DOM. For a standard solution, we want to tackle the core problem head-on.

As mentioned in the previous section, we want to make features like find-in-page work with the built-in virtual scroller. We have briefly considered adding a set of find-in-page APIs to the web platform, that would support cases like giving the web author a way to completely override the find-in-page command, or interacting with and adding results to the user agent's built-in find-in-page functionality.

However, designing these APIs proved to be quite challenging, given the breadth of find-in-page user interfaces across browsers. Worse, providing a find-in-page-specific solution might unintentionally become a disadvantage for other things like accessibility: web developers might be inclined to think that a virtual scroller that works with find-in-page is good enough, and not think about the remainder of the missing functionality caused by virtualization.

Libraries

Another approach would be to standardize and implement only the low-level primitives which allow mitigating the cost of DOM, i.e. display locking. We would then leave the building of high-level virtual scroller APIs to libraries.

We fully expect that some applications and libraries will take this route, and even encourage it when appropriate. But we still believe there is value in providing a high-level virtual scroller control built into the platform, for the 90% case. For more on our reasoning, see the motivation document's "Standardization" and "Layering" sections.

Sample implementations

Chrome

Launch chrome with flags --enable-blink-features=DisplayLocking,BuiltInModuleAll to get a working virtual-scroller element.

Demos

https://github.com/fergald/virtual-scroller-demos

virtual-scroller's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

virtual-scroller's Issues

use IntersectionObserver?

VirtualList computes the viewport manually by diffing the scrollTarget bounds with the container bounds. We use getBoundingClientRect() to get scroll position and size, so we force layout.
We might be able to use IntersectionObserver to get the scroll position info w/o causing relayout.
For the size, we can rely on ResizeObserver (#16)

ItemsSource API idea

I'd like to propose the following API refinement. Currently you do

list.newChild = index => childElement;
list.updateChild = (child, index) => ...;
list.recycleChild = (child, index) => ...;
list.childKey = (index) => key;
list.totalItems = length;
list.requestReset();

I think we should introduce the concept of an "item source" which translates indices into items, like so:

list.newChild = item => childElement;
list.updateChild = (child, item) => ...;
list.recycleChild = (child, item) => ...;

list.itemSource = {
  init(list) { /* see below */ },
  item(index) { return item; },
  get length() { return length; },
  childKey(index) { return key; }
};

list.itemsChanged();

So far this is not very different:

  • It groups things related to the items together
  • It lets the newChild/updateChild/recycleChild functions go back to operating on items instead of indices, which is somewhat nice
  • It changes from a totalItems setter to a getLength() function, which the implementation calls lazily. This means that you can no longer trigger updates by setting totalItems, but must always call itemsChanged() instead.

But this enables us to provide some helpful item source factories:

list.itemSource = new ArrayItemSource(contacts, c => c.name);

contacts.push({ name: "Domenic" }); // doesn't do anything
contacts[1].name = "Valdrin";
list.itemsChanged();                // Let the list know to refresh from the item source
ArrayItemSource implementation:
class ArrayItemSource {
  constructor(array, childKey) {
    this._array = array;
    this.childKey = childKey;
  }
  
  item(index) {
    return this._array[index];
  }
  
  get length() {
    return this._array.length;
  }
}

or even

list.itemSource = new SmartArrayItemSource(contacts, c => c.name);

list.itemSource.push({ name: "Domenic" }); // automatically updates the list!

// But, this still needs to be done manually:
list.itemSource[1].name = "Valdrin";
list.itemsChanged();

// But this would work automatically:
list.itemSource[1] = { name: "Valdrin" };
Not-working-yet SmartArrayItemSource implementation...

Something's off about trying to add methods here... I think I need to override get() too or something. But, you get the idea: it's a subclass of Array with some extra methods to also make it conform to the item source interface, including init() which stores the list.

class SmartArrayItemSource extends Array {
  constructor(array, childKey) {
    const normalProto = new.target.prototype;

    const wrapped = new Proxy(array, {
      set(...args) {
        const returnValue = Reflect.set(...args);
        if (array._list) {
          array._list.itemsChanged();
        }
        return returnValue;
      },
      getPrototypeOf() {
        return normalProto;
      }
    });
    
    wrapped.childKey = childKey;
    
    return wrapped;
  }
  
  init(list) {
    this._list = list;
  }
  
  item(index) {
    return this[index];
  }
}

let's discuss render timing expectations

Context: see this note from @atotic on ResizeObserver timing (rAF, then ResizeObserver, then Paint)

Currently we batch rendering with a rAF. This means that a user can expect to have the virtual-scroller rendered by the next frame.

Is it reasonable to explicitly say this

expect virtual-scroller to be done rendering by the next rAF (1)

vscroller.totalItems = 10;
requestAnimationFrame(() => {
  console.log(vscroller.childElementCount);
});

or should we instead say

expect virtual-scroller to be done rendering by the next paint (2)

vscroller.totalItems = 10;
requestAnimationFrame(() => 
  Promise.resolve().then(() => {
    console.log(vscroller.childElementCount);
  })
);

With option (1) we're currently obliged to force reflow on the container, as ResizeObserver callback is invoked after the rAF callback.

Option (2) would allow us to use the ResizeObserver timing as signal to render synchronously. I don't know if there is a "blessed way" for having an afterPaint callback, so we could expose a afterNextRender promise in virtual-scroller to be used exactly for this:

vscroller.totalItems = 10;
vscroller.afterNextRender().then(() => {
  console.log(vscroller.childElementCount);
});

@domenic WDYT?

Type of items property

Right now the items setter just takes whatever its given and stores it. The getter returns the stored value.

The more web-platformey thing to do would be:

  • The setter accepts any iterable, snapshotting that iterable at the time it's set
  • The getter returns a frozen array of the snapshot

The main benefit here is that you can't write code like

list.items.push(...);

because the array is frozen. (Such code won't work, because there's no notification that the array changed. You'd instead need to use requestReset(): see #26.)

The drawback of this is that it's O(n) in the total number of items. The current requestReset() pattern allows O(1) item additions and removals.

Not sure what to do here, but I wanted to have a tracking issue open.

Setter-only design vs. constructor arguments

https://github.com/valdrinkoshi/virtual-list/blob/design/DESIGN.md has a lot of examples where you construct various classes by creating empty versions of them and then setting a bunch of properties with Object.assign(). This is strange in a few ways:

  • It implies that the objects are fully functional even before these properties are set. Is that true? I find it hard to believe that a VirtualList with no layout or items is still a fully-functional VirtualList.
  • It implies that these properties can be changed at any time and the class will respond appropriately. (E.g., changing container.)
  • It implies that these properties can all be changed independently of each other. (Maybe this is true, I'm not sure.)

Instead I would suggest making these constructor arguments: new VirtualList({ layout, container, items, newChildFn }), etc. This ensures that you cannot construct an instance of the class without setting them, and setting them all at once, together.

This also allows more fine-grained separation so that things which can be set and modified later and independently (say, VirtualRepeater's first?) are exposed as public getter/setter pairs.

rename <virtual-list> to <virtual-scroller>

<virtual-scroller> is the thing that needs to be sized and creates DOM as the user scrolls to it

<virtual-content> is the thing that auto-sizes itself according to the estimated size of the items, and creates DOM only for the visible portion of it (can be contained in any scroll container).

should we ship requestReset()?

<virtual-list> currently exposes the requestReset() method as a mean to trigger a rerendering of the DOM. This is useful for users that want to keep the same items array instance, perform an arbitrary number of updates to it, and decide when to trigger the rerendering.

Should we expose such method?

Other possible solutions:

  1. always trigger the rerendering when items is set
  2. require the user to provide a new items array instance to trigger the update

If we want to keep this method, what would be a better name for it?

Terminology for x/y and width/height

We should stay consistent with the rest of the platform, and general use these nouns:

  • top
  • left
  • width
  • height

The current design sometimes uses "y" instead of "height" (e.g. in itemSize) or "y" instead of "top" (e.g. in scrollTo).

"Fn" and "Callback" suffixes probably not needed

In general we can just use an name for the method, without suffixing it with something indicating its type. Custom elements have a weird "Callback" suffix but that is because of worries about collision with other things on element subclasses; it's not generally a good pattern.

different template engines (lit-html/preact)

<virtual-scroller> expects new/update/recycleChild methods to return/handle Element nodes, and VirtualRepeater relies on Element native APIs to append/move/remove/measure these.

Other templating engines like lit-html or preact create different types of instances for their rendering purposes, e.g. lit-html deals with TemplateResults, preact with VNodes, and their render() methods know how to convert these into DOM to be stamped in a container.

The current virtual-scroller makes it confusing for the user to deal with both TemplateResults/VNodes and Elements. E.g. this is the most ergonomic way to use virtual-scroller with lit-html/preact http://jsbin.com/secetor/1/edit?html,output

import './virtual-scroller-element.js';
import {html, render as litRender} from './node_modules/lit-html/lib/lit-extended.js';
import {createElement, render as preactRender} from './node_modules/preact/dist/preact.esm.js';

const wrapper = 'div';
const totalItems = 100;

/* -- raw -- */
const rawScroller = (totalItems) => {
  const scroller = document.createElement('virtual-scroller');
  scroller.newChild = (i) => document.createElement(wrapper);
  scroller.updateChild = (el, i) => el.textContent = `index ${i}`;
  scroller.totalItems = totalItems;
  return scroller;
};
const rawRender = (elem, container) => container.append(elem);

rawRender(rawScroller(totalItems), rawContainer);

/* -- lit-html -- */
const litScroller = (totalItems) => html`
    <virtual-scroller
      newChild="${() => document.createElement(wrapper)}"
      updateChild="${(el, i) => litRender(html`index ${i}`, el)}"
      totalItems="${totalItems}"></virtual-scroller>`;

litRender(litScroller(totalItems), litContainer);

/* -- preact -- */
const preactScroller = (totalItems) => createElement(
    'virtual-scroller', {
      newChild: () => document.createElement(wrapper),
      updateChild: (el, i) => el.textContent = `index ${i}`,
      totalItems,
  });

preactRender(preactScroller(totalItems), preactContainer);

How can we make this more ergonomic for the lit-html and preact users? How would that look like?

Both lit-html and preact need to generate instances that are not Element nodes, and to handle how these instances are rendered via their specific render() method, whereas virtual-scroller needs to keep track of the created children, and to append/move/remove/measure them.

@justinfagnani @developit @domenic WDYT?

Render fixed number of items (for SSR / SEO)

A virtual-scroller is great for infinite scrolling but for SEO purposes it's good to make the pages of data addressable as described here:

https://webmasters.googleblog.com/2014/02/infinite-scroll-search-friendly.html

(and the rangechange event in virtual-scroller makes this much easier to implement! 👍)

Would it be useful to be able to force the n (page size) items to be rendered when in a SSR / bot scenario, regardless of the height of the list container and how many items might otherwise be rendered?

I know it would be possible to just switch to a different template and render things as a plain list but wondered if it might be easier / simpler to just force some fixed item render count as a parameter.

child templating: init method instead of setter?

In #13 we moved from setter-only design to constructor arguments.

In <virtual-list> element we chose composition over inheritance. We lazily init a VirtualList once the user provides the newChild method through the <virtual-list>.template setter.

Rather than having the template setter, we could introduce an init(config) method where the user can pass the child templating methods, e.g.

document.querySelector('virtual-list').init({
  newChild: (item, idx) => {
    const child = document.createElement('div');
    child.textContent = idx + ' - ' + item;
    return child;
  },
  // updateChild: ...,
  // recycleChild: ...,
  // direction: 'horizontal',
  // items: []
});

padding <virtual-scroller> requires sizing children accordingly

<virtual-scroller> positions children with position: absolute and sets their width: 100%, so if the virtual-scroller has padding, children would effectively get the total width of their container (padding included). The user would have to handle that e.g. like this:

virtual-list {
  padding: 10px;
}
virtual-list > * {
  width: calc(100% - 20px);
}

See if we can do something to handle padding too, or at least document this limitation

consider exposing childSize(child, index)

We currently measure each child's bounds and margins during rendering, this tho could be provided by the user who knows the size, making the list rendering even faster.

childSize(child: Element, index: number) => { width, height, 
                                              marginTop, marginBottom,
                                              marginLeft, marginRight }

Here an example usage:

virtualList.newChild = (index) => {
  const el = document.createElement('div');
  el.style.height = '50px';
  el.style.marginTop = '10px';
  el.textContent = index;
  return el;
};

virtualList.childSize = (child, index) => {
  const size = { height: 50 };
  if (index !== 0) size.marginTop = 10;
  return size;
};

virtualList.totalItems = 10;

<virtual-list> styled with `contain: strict`

Currently <virtual-list> styles its host with contain: strict to help the browser reduce the layout/paint/style computations.

One of the drawbacks is that content intentionally overflowing outside the virtual-list will be clipped, e.g.

<style>
  virtual-list {
    width: 200px;
  }
  .tooltip {
    position: absolute;
    top: 0;
    left: 180px;
  }
</style>
<virtual-list id="list"></virtual-list>
<script>
  list.newChild = () => {
    const child = document.createElement('div');
    child.innerHTML =`
     <span>Hello world!</span>
     <span class="tooltip">custom tooltip</span>`;
   return child;
  };
  list.items = new Array(100);
</script>

The user can still override the default value via css, e.g. virtual-list { contain: none; }, and we could also expose a css custom property.

Before doing that, is it worth for <virtual-list> to set this default style?

how to detect scroll initiator?

In #83 we'd like to know when the scroll event was caused by user action vs when it was caused by resize events.

We want to invalidate anchoring information when user scrolls, and compute (if needed) and use the anchoring information on size changes.

For most of the cases diffing the previous scroll position with the new one is enough. This tho doesn't work in the following case: the user scrolled at the bottom of the list, and now resizes the window to make it larger:
=> the items occupy less height
=> the container scrollHeight decreases, hence the browser adjusts the scrollTop to be scrollTop = scrollHeight - offsetHeight
=> this provokes a scroll event
=> the diffing of the previous scroll position with the new one considers this as a user scroll 😭🔫

Here a simplified example where we change the size of items above the last one http://jsbin.com/vejuzo/1/edit?html,output

The only way to handle this i can think of is to ignore the scroll events when we detect a size change, and to stop ignoring them after a setTimeout(fn, 1) 🤢

Edit: Another idea is to adjust the scroll position when we receive the new viewport size - so that the diffing will result in no changes 💡

@domenic @graynorton any ideas? This seems a good candidate for missing platform features maybe.

[feature] expose itemKey function

The Repeats mixin keeps track of the created DOM in the key/child map _keyToChild. When a child is not found, it will invoke newChild to create it and stores it in the _keyToChild map.
In this example newChild is invoked only for the newly added item, while updateChild is invoked for all the items:

list.items = list.items.concat({name: 'new item'});

_keyToChild uses index as the key. This can be customized by providing the itemKey function to return a custom key itemKey(item: any) => string|number

Should we expose this property in <virtual-list>?

Let's discuss microtask-batched reading of items array

Per #48 (comment), apparently virtual-list only reads from the items array after a microtask delay. This is pretty unusual, and I think we should discuss it.

In particular, the example discussed there is pretty confusing. Consider a list where you have only set newChild. If you do

// (1)
list.items = x;
list.items = y;

then it will call newChild for all the items in y. But if you do

// (2)
list.items = x;
setTimeout(() => {
  list.items = y;
}, 0);

Then it will call newChild for all the items in x, and the items in y are ignored. I would prefer that both are consistent: i.e. both do newChild for all items in x.

Now consider a list where you have set both newChild and updateChild. Then (1) will call newChild + updateChild for all the items in y, and (2) will call newChild for all the items in x, and updateChild for all the items in y. I would prefer that both are consistent: i.e. both do newChild for all items in x, and updateChild for all items in y.


Note that the platform has some precedent for delayed rendering. img.src = x; img.src = y; will only try to load y. This case feels worse because of the newChild/updateChild distinction, though.

MutationObservers are also microtask-batched, but they don't do coalescing, so they tell you all the things that happened since the last time, not just the latest thing.


I'm open to being convinced that batching is more important than consistency. But I lean toward saying that if developers change items a lot, they should know that will cause a lot of updates, and we shouldn't try to batch for them.

Maybe if developers want batching, we should have something more suited to their use case, e.g.

const resumeUpdates = list.suspendUpdates();
... mutate list.items a lot ...
resumeUpdates();

Also, if we do think batching is important, I think microtask batching is probably not right, and we should do requestAnimationFrame batching instead.

Thoughts?

Rename requestReset

As you need to set updateElement before calling requestReset, wouldn't requestUpdate not be a more explaining name? It would also make it more clear that you had to implement updateElement.

invalidate would be another suggestion, though that doesn't show the link with updateElement.

request to me sounds like something that might be rejected. scheduleUpdate? Maybe that is why I like something like invalidate

jumpy scrollbar with VirtualList

When the list items have variable height, the user can notice a "jumpy" scrollbar behavior while scrolling.
This is caused by the error adjustment done by the Layout. This should be mitigated by memoizing the sizes.
e.g.

import Layout from './layouts/layout-1d.js';
import {VirtualList} from './virtual-list.js';

const vlist = new VirtualList();
vlist._recycledChildren = [];

Object.assign(vlist, {
  items: new Array(20).fill({name: 'item'}),
  container: document.body,
  layout: new Layout(),

  newChildFn: (item, idx) => {
    let section = vlist._recycledChildren.pop();
    if (!section) {
      section = document.createElement('section');
      section._title = section.querySelector('.title');
      // Update it immediately.
      vlist._updateChildFn(section, item, idx);
    }
    return section;
  },

  updateChildFn: (section, item, idx) => {
    section.style.height = (100 * (idx + 1)) + 'px';
    section.style.outline = '1px solid';
    section.id = `section_${idx}`;
    section.textContent = `${idx} - ${item.name}`;
  },

  recycleChildFn: (section, item, idx) => {
    vlist._recycledChildren.push(section);
  }
});

html-spec-viewer breaks styles

The html spec stylesheet styles elements with selectors like body > p, hence hosting the content inside <html-spec-viewer> breaks these styles.

Consider not loading layouts dynamically?

Right now the code loads layouts dynamically. I'm not sure that makes much sense for something that would be bundled with the browser.

But maybe it does? After all, you still pay some cost for loading the layouts in to memory, and maybe you should only pay those costs for layouts you use. I'm not sure.

From a spec perspective, I guess the impact of loading layouts lazily is that we'd have to mandate in the spec that changing the layout can take an arbitrarily long time to apply (e.g. if loading them takes longer than one animation frame). That seems a bit bad.

[feature] set scrolling element

Currently <virtual-list> supports only the main document as scrolling element. Ideally we'd want to allow the user to setup another scrolling element.

We could check the css overflow property of the container, and if it is scroll or auto then the container is the scrolling element

Provide better explanation for updateChild and recycleChild

newChild function is invoked when the <virtual-list> needs DOM to represent an item.

recycleChild function is invoked when the <virtual-list> has already set display: none on the child and is about to remove it from the document. if not defined, <virtual-list> will remove the child from the document. The user can use this property to stop the default behavior and manage the DOM.
When defined, <virtual-list> will not remove the child from the document to avoid a removeChild/appendChild sequence for performance reasons. The user can still decide to remove the child from the document in the recycleChild function.

updateChild function is invoked after the <virtual-list> has appended the child to the container and requires it to be updated with data.

  1. Is there a better name that conveys the intent of these functions?
  2. shoulde we fire an event instead of recycleChild? Something like willremovechild? We'd still need a way to know that it was handled (e.g. via preventDefault()) in order to prevent the removal of the child from the document.
  3. do we really need updateChild? the user could just use newChild and recycleChild
const vList = document.querySelector('virtual-list');
const pool = [];
Object.assign(vList, {
  items: new Array(20).fill('item'),
  newChild: (item, idx) => {
    const child = pool.pop() || document.createElement('div');
    child.textContent = idx + ' - ' + item;
    return child;
  },
  recycleChild: (child, item, idx) => {
    pool.push(child);
  }
});

^ We need updateChild for cases like data updates w/o changing first, num

Protected methods/properties using _ syntax

A pattern of "protected methods" that are prefixed with _ is not present on the web platform.

Instead, I'd suggest making these kind of customizations constructor arguments, similar to #8. You can see precedent for this in various places on the platform, such as promises and streams.

I might not fully be understanding these though, as I don't see why you used one design for _measureCallback and one for updateChildFn.

rename itemKey to childKey

itemKey(item: any) => any is used to determine an identifier for the DOM corresponding to the array item.
We should probably rename it to childKey, as it is effectively a sort of "id" for the stamped dom.

We could name it childId, but that might lead users to think that the generated dom has an id attribute.

Problems loading dynamic imports in rolled-up bundle

I had to change virtual-list-element.js:167 in order to load modules when bundling with rollup.

patch-package
--- a/node_modules/virtual-list/virtual-list-element.js
+++ b/node_modules/virtual-list/virtual-list-element.js
@@ -164,7 +164,7 @@ export class VirtualListElement extends HTMLElement {
     Object.assign(list, {newChild, updateChild, recycleChild, itemKey, items});

     const Layout = await importLayoutClass(
-        this[_grid] ? './layouts/layout-1d-grid.js' : './layouts/layout-1d.js');
+        this[_grid] ? '/node_modules/virtual-list/layouts/layout-1d-grid.js' : '/node_modules/virtual-list/layouts/layout-1d.js');
     const direction = this[_horizontal] ? 'horizontal' : 'vertical';
     if (list.layout instanceof Layout === false ||
         list.layout.direction !== direction) {

See #27

recycleChild called too late

When scrolling, recycleChild gets called after newChild, so the user would collect children too late to be reused for that render loop.

This seems caused by the fact that we enter in _incremental mode during rendering, which doesn't invoke recycleChild but does always invoke newChild.

`will-change: transform` on the virtual-list children by default?

<virtual-list> positions its children with position: absolute and transform: translate3d.
Should it also style the children with will-change: transform by default?

There are some implications for animations, e.g. see section "Partial solution: set will-change: transform" here https://greensock.com/will-change
This might be mitigated by the fact that we need to position children with position: absolute, hence we're creating a stacking context for each.

subpixel positioning causes text aliasing/blurriness

VirtualList positions children via translate2d with values that have subpixel precision. This seems to cause some weird artifacts like blurriness (seen on Canary on the html-spec-viewer)

Figure out if need to avoid subpixel or we can use some magic css property.

Requirements checklist

Tracker for the features to be implemented - copied from https://github.com/domenic/infinite-list-study-group/edit/master/REQUIREMENTS.md.

Mark complete after providing implementation and documentation.

P0

  • Accurate scrollbar position for elements that have been loaded
    • Implemented, see demo/layout.html (scroll, change items count, and observe scroll position is kept)
  • Overall list structure supports screen readers and accessibility controls
  • Heterogeneous elements
  • Fallback element UI
  • Grid layout
  • List headers/footers
  • Configurable layout and style for elements in the list
    • User can style content via virtual-list > * or by adding arbitrary classes to the stamped children
    • layout configurable via the layout property
  • Clickable list elements
    • Controlled by the user via newChild/updateChild properties
    • example in the README.md
  • indexability of not loaded content

P1

P2

  • Backed by remote data source
    • demo/fallback.html shows how contacts are lazily loaded while we place a fallback during the loading
  • Selectable list elements
    • TODO demo/selection.html
  • Rearrangeable list
  • Find in page (ctl+F) highlights all elements in the list that haven't been loaded
  • Accurate scrollbar position including elements that haven't been loaded
  • Pull to refresh
    • TODO demo/pulltorefresh.html
  • Discoverable by screen readers for elements in the list that haven't been loaded
  • anchor refs pointing to dom not yet rendered
  • Horizontal scrolling
    • <virtual-list layout="horizontal"></virtual-list>
    • see layout property in the README.md

items (array) vs totalItems (number)

Neither VirtualRepeater nor VirtualList actively use items or mark in any way the objects in the array, so it should be possible to setup a list with only totalItems (number) instead of items (array). This would translate in a setup like this:

const items = new Array(10).fill('item');

/* old setup with items (array) */
new VirtualRepeater({
  items: items,
  newChild: (item, idx) => document.createElement('div'),
  updateChild: (child, item, idx) => child.textContent = `${idx} - ${item}`,
  childKey: (item) => item.key
});

/* new setup with totalItems (number) */
new VirtualRepeater({
  totalItems: items.length,
  newChild: (idx) => document.createElement('div'),
  updateChild: (child, idx) => child.textContent = `${idx} - ${items[idx]}`,
  childKey: (idx) => items[idx].key
});

One drawback of this approach is that we cannot check anymore for identity in the totalItems setter, meaning that we'd have to always reset the list when totalItems is set, whereas with items we could check for strict identity in order to trigger the reset.

default values for newChild and updateChild

What about setting these default values for newChild and updateChild?

constructor() {
  super();
  this.newChild = () => document.createElement('div');
  // assuming we have itemsSource API
  this.updateChild = (child, item) => child.innerHTML = item.toString();
}

The user can still override these values to customize the scroller.

A default newChild helps especially those template libraries that need a container to render within, e.g. lit-html and preact. With lit-html, the user would just need to do this to update the nodes:

import {render, html} from './node_modules/lit-html/lib/lit-extended.js';
import './virtual-scroller-element.js';

const itemTemplate = (item) => html`item ${item.someproperty}`;
const updateChild = (child, item) => render(itemTemplate(item), child);

const scrollerTemplate = (totalItems) => html`
   <virtual-scroller updateChild=${updateChild} totalItems=${totalItems}>
   </virtual-scroller>`;
      
render(scrollerTemplate(1000), document.body);

What about 2-D structures?

If we want wide adoption of this pattern, it'd be nice if there was a way to extend it to at least 2 dimensions - think something like a spreadsheet or a map of tiles or some other virtually infinite 2-dimensional structure.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.