Re-building core to handle progressive data updates about chartist HOT 42 OPEN

Hi there. Chartist is not yet optimized for realtime data which means a complete SVG DOM reconstruction on data updates instead of a mechanism to update the existing DOM. This can quickly lead to performance problems. But I started an angular module that provides directives for charts which is doing realtime data visualization (although in a very unperformant way) and i can possibly share this in a repo so you can take a look.

I believe that "realtime data" would be a feature request that needs optimizations in the core so ill flag this issue as such.

from chartist.

👍 I'm hoping to start digging into this library soon, and I'll be interested in checking out how progressive updating might work.

from chartist.

@gionkunz would you mind publishing the angular wrapper you're working on? I'd be interested in contributing.

from chartist.

Hi

I'll publish it asap. Have to fix other stuff first.

Cheers
Gion

from chartist.

👍 to realtime improvements.

from chartist.

@gionkunz is paradox41/ng-chartist.js the wrapper you were talking about?

from chartist.

@marshall007 no this is @Paradox41's work there 😃 . But I'm fully supporting this efforts and will also contribute in his repository. I've already included a few things from the original PoC I was running offline on my machine.

from chartist.

In fact, thanks to @gionkunz's contributions, there is an example there with progressive data updates.

from chartist.

Hi Guys

I'm thinking about this now more in detail. The problem currently is that we recreate the whole DOM on an update. However, thinking of a way to support data manipulation and progressive updates, it's really a tricky thing. The problem currently is that manipulating data in a chart that is equally distributing the graph elements along the axis is nearly impossible without a complete redraw. Let's say we add some data at the end. Usually progressive mechanisms benefit from moving blocks around without the need to modify them. However, no matter where we add data, we will need to go though each and every entity on the chart and update the coordinates because we have less space available to distribute the entities. This means, even if we don't recreate the dom, we still need to update the dom, which will be, I fear, almost as expensive.

Check out this example: http://jsbin.com/lebox/edit?js,console,output

It shows the fps when you do a update cycle every frame. We could now assume that we can get a few more FPS out of chartist if we only update the dome and don't recreate it, but my guess is that we will face a different limit very quickly.

I believe that for real time charts we would need to have a different approach where we scroll through the chart automatically so we don't need to update the distance between the entities.

An other approach would be to use % in the SVG path and let the browser rendering pipeline take care of the re-arangements. However, we still would need to update the percentage values.

For updating chart values that will not add or remove any values I can clearly see the benefit of not re-creating the dom.

Maybe this whole thing makes sense with a note that changing the count in the data will cause performance issues?

Do you have other thoughts?

Cheers
Gion

from chartist.

Maybe this whole thing makes sense with a note that changing the count in the data will cause performance issues?

Maybe we can have an option like maxItemCount and chartist would then keep only the last X items.

from chartist.

Yeah, that could be done. @Paradox41 what's your input here?

from chartist.

It is an interesting problem. My first instinct would be to see how other charting libraries that use svg handle updates - e.g., dc.js. Obviously, whatever work we can offload to the browser would be ideal.

Do you have any idea on how you plan to track changes in data sets? I feel like redrawing is something that can be optimized as long as the bookkeeping is being done in a sane way.

I've been trying to find time to dig into the source code here, but work has been busy. :(

from chartist.

Hey Will. Yeah I was looking at dc.js too. The thing is that the only moment when there is a change in data points on their example is when you zoom in on the line / area graph. This is also the point where the FPS drops down to 0 for a moment :-) Most of the transformation is done using D3 and it does animate the SVG attributes with javascript (not SMIL as far as I could see). So its just a matter of animating the real value down to 0. I'm not concerned if this is the only goal here because we can also just do a attribute change, but still there is no way around DOM manipulation.

I think for now we could just introduce a data manipulation API that allows to manipulate the data. If there are no changes in the count we can do a simple modification and if there are changes we anyway need a some DOM creation.

I think I'll create some performance tests with some DOM recycling approaches when doing simple transformations like modification, delete and add data. Based on that we can figure out the best way to do it in chartist. Feel free to create some tests as well.

One other thing I was working on which will increase performance a lot is the way how Chartist creates labels. You can track this here #25 and I'll need some feedback there too! But this will give already a huge performance boost.

from chartist.

I was trying to do this myself just now with Angular, and noticed it was re-constructing everything. For example, I am using a bar chart with 12 months and I wanted to slowly animation each bar from its old value to its new value. I'll check out the ng wrapper real quick and see how that works out.

from chartist.

@moneytree-doug if you like to do animations check the smil animation feature that was introduced with 0.4.0

from chartist.

Look into React https://github.com/facebook/react
It optimizes the DOM rendering and re-rendering and already supports SVG to some extent.

from chartist.

What if you draw a new svg offscreen and then replace the current one? In the case of animations, you can set the initial location at their current values, replace svg element, then smil to the new values. Maybe I'm oversimplifying :)

from chartist.

@luisfrocha drawing offscreen is better for sure as you don't cause reflows but still there is a massive overhead using real DOM objects. The solution of using a virtual DOM is preferred and was discussed here: #87

from chartist.

@gionkunz a virtual DOM meaning something like React or something like server-side rendering? The issue with SSR is that you have to indeed run a server like Node or something like that, whereas right now Chartist can be run with just a static site.

from chartist.

@ilanbiala Virtual DOM is an abstraction nessesary to implement something like React and server-side rendering.

Imagine you want to display some text. When in browser, you make a DIV with document.createElement, but you don't have document.createElement on the server. To use the same code for displaying text on the browser-side and on the server-side you have to either implement document.createElement on the server-side or change your code to call some displayText function that uses document.createElement in the browser but uses something else on the server.

from chartist.

@gionkunz if we start with offscreen rendering, how much does that improve performance for each chart type?

from chartist.

progressive data updates +1

from chartist.

progressive data updates +66356

from chartist.

@gionkunz please take a look at http://oboejs.com/; ideally a progressive solution should be able to work seamlessly with Oboe.

from chartist.

Any updates on progressive updating? Would be amazing!

from chartist.

@logitimate This already works using the update function (check the documentation). This ticket is still open because the current update is a full render which implies bad performance (although it's pretty fast) and the missing possibility to animate the updates.

from chartist.

Wondering if it even needs to be passed as an option at all (as suggested by maxItems).

For example, if I pass a method to a line chart (something like addLine(data, options)), it would take the array passed to that function and map it onto the existing chart. This method could check the number of data points on the existing chart (say x) and map only x number of values from the array. All other values would be discarded.

Of course this puts the responsibility on the dev to ensure the same number of points are being passed at all times, but that's a risk I would gladly accept to achieve this functionality.

In my use case, I'm always passing the same number of items to the line graph (including 0) to show data over time.

from chartist.

@lukad03 What's wrong with using the update function ? http://gionkunz.github.io/chartist-js/api-documentation.html#chartistbase-function-update

from chartist.

Well, I guess you want animations... :-/ But for such use cases we could also have a plugin that keeps track of the data changes and if it detects that the amount has not changed it could create an animation from the previous locations to the new ones. Actually I'm really trying to think about the radical refactoring that we would need to go through in order to include progressive data updates and the resulting benefits. For animation I'm sure we could work with a plugin and in terms of drawing performance, as already statet earlier, updating 1000 attributes is maybe just a bit faster than rendering 100 element.

from chartist.

I know that React uses shadow DOMs to keep track of changes to reduce paint problems. Maybe that could be a method to test, although probably not done very easily.

from chartist.

I guess you meant virtual DOM right? Yeah we've also discussed that but the problem is the amount of changes. For an animation or just introducing a new value, all other data points and grids will change which even with virtual DOM is almost equal to a full repaint.

from chartist.

@gionkunz Yes, animations are what I'm seeking :D. Ideally you could control whether an existing line animates to a new form, or add a new line to the graph that wasn't there before without having to redraw the entire chart.

from chartist.

I'm working on something to extend chartist with a custom updater / renderer.

Using chartist event system I was able to listen for data changes and compare what changed on the data side and only update the UI based on the diff between the old data and the new data.

This doesn't require redrawing the whole UI, it only requires updating the data points that changed.

By indexing the data points, we can use the index from the data to locate the UI element to update, requiring only one redraw per update. If 10 data points changed, then we can make all 10 changes at once. This works because the series data points are a 1:1 mapping of the data and the number of points being displayed on the UI, i.e the UI has the same number of elements as the data does.

Other improvements can be made like querying the DOM less often. It is often beneficial to convert a live nodelist to an array as it will perform better than a static nodelist, i.e a nodelist that is acquired with querySelectorAll() is a static nodelist.

See here http://jsperf.com/nodelist-vs-array-iteration for more details.

Currently I think there are two ways that will make the chart "realtime":

• Streaming data: this means that data can be streamed at the end, i.e pushed at the end of an array and we only take the last maxItemCount as suggested above. Old data is automatically flushed out of the array, i.e shifted out of the array. This way only new data points are "streamed" in. Redraws then occur based off of the last streamed data. It is easier to work with smaller data, i.e a stream because the redraw doesn't have to re-examine every data point. The size of the stream will be limited by maxItemCount this ensures faster computations, as the array will have a smaller fixed size length.
• Full refresh: the whole data array is replaced with a new array, this will call a redraw for all data points. I think this will be more expensive on the UI.

Those are my thoughts, I will have an example in a couple days as I'm still building out the UI updater.

from chartist.

How would you handle additional data points? Each update will cause the path attribute to change on line charts and if only one data point gets added or removed, youd need to reposition all other data points. Progressive data update would certainly support animation but not performance.

from chartist.

Streaming data solves the additional data points, for example if your chart has maxItemCount of 10, it is much easier to work with 10 points then say 1000 points.

I did something similar with c3.js - a d3 charting library. Naturally this library does not have realtime charting, but it does react to data changes.

See my codepen for a real example of how the data points is kept to a minimal and updates happen accordingly.

Take a look at the chart on the right, this is how I imagine streaming new data in.

http://codepen.io/abacaj/pen/PPXrNQ

from chartist.

I'm using this library (it's awesome) for a private project and I'd love it to implement streaming (+animations) like highcharts do - meaning automatically adjusting axis etcetera.
http://www.highcharts.com/studies/live-server.htm

from chartist.

Is there any news on this? Can we somehow help with this?
I got a working example with react https://medium.com/@sakulstra/meteor-mqtt-live-plotted-graph-de95773e12d2#.hmrjmu7g8 , but it doesn't animate... also I hacked around to only update when the tab is active. This could be somehow built in I think, as rendering without viewing isn't that clever ;)

from chartist.

Hmm, how does Google do it? https://developers.google.com/chart/interactive/docs/animation

from chartist.

Well they repaint the whole chart for their animations ;-) Not very CPU friendly. Just check the CPU profile in your debugger for this page.

from chartist.

I think we'll need to build streaming and animation in the core which is probably not going to happen in the 1.0.0 release as the milestone suggests. :( I think this will need to go to version 2.

from chartist.

@gionkunz won't using a virtual dom implementation like snabbdom be more efficient even if it requires a full repaint? Another advantage is that one wouldn't have to call specific chartist function for updating a graph etc.

Maybe the idea is to do a separation of concerns from generating the SVG from the calculation of the co-ordinates for the vectors/shapes etc so that any backend for generating SVG (current direct DOM calls/virtual dom library can be plugged in?

from chartist.

It'd be great if render instructions were passed in to an optional user provided render function as data. That way you can take those instructions and render in React/Vue/mithril/cycle/snabbdom etc.

I'm sure the community would fill in the gaps and release specific modules for different ecosystems. That lets those libraries that have already optimized diffing svg handle it, and hopefully without breaking the existing API for static data.

The instructions themselves could be hyperscript output { tag, attrs, children}[], or something more specific to charting. Interop for a particular library could be as easy as instructions.map( React.createElement )

from chartist.