Loading Data Driven parameters takes a long time for significant data sources about extension-data-driven-parameters HOT 23 CLOSED

Right now I'm in proof-of-concept phase so paring the list down will be done by using a 'top' clause in the custom SQL. The entity being selected has a large number of properties that can be used to restrict the list in a production situation, but that will involve the actual client; we're not there yet.

from extension-data-driven-parameters.

While I'm asking, could you make the filter for duplicates an option? (My data is already "clean" due to it being a unique index.)

Also, could 'refresh on select' be optional as well? If it's going to take a while to load, I wouldn't want the user to incur that after every selection.

from extension-data-driven-parameters.

Hi @JohnDelahunt,

I've added the option to not reload the list to the extension under "Update list of values on parameter change.". As this was the default behavior you'll have to re-configure the extension and un-check it to see the results. Let me know if this helps.

As for the sorting and uniqueness, would you be willing to try these changes on your own by removing those parts in the code? If you see improvements without those turned on I will gladly add them to the extension options so you can still remain up-to-date.

Take care,
Keshia

from extension-data-driven-parameters.

Had some issues with the new updates so I rolled them back until we can confirm everything is good. But here is an alpha with the updates. Please do not use the alpha in production as it can change or be removed any time but let me know how it works for you and if you come across any bugs!

from extension-data-driven-parameters.

The option to not reload on selection works as expected.

I'm still hoping you can do something about the speed of the initial load. I'm assuming the sorting and filtering for duplicates are the parts of the process where the time is spent. Neither of these is necessary when the data source takes care of them already.

from extension-data-driven-parameters.

Hi @JohnDelahunt,

I've added the option to not reload the list to the extension under "Update list of values on parameter change.". As this was the default behavior you'll have to re-configure the extension and un-check it to see the results. Let me know if this helps.

As for the sorting and uniqueness, would you be willing to try these changes on your own by removing those parts in the code? If you see improvements without those turned on I will gladly add them to the extension options so you can still remain up-to-date.

Take care,
Keshia

Hi @KeshiaRose,

I’m also looking into this possible improvement and would like to take a stab at trying this. Can you point me to which file/part of the code should be removed so I can try it on my end?

Thanks,
Kim

from extension-data-driven-parameters.

Hi @kimmte,

In DataDrivenParameter.tsx comment out lines 159, 165-169, and 175-179. Then run the application as described in the readme. Let me know if you have any issues testing locally!

Take care,
Keshia

from extension-data-driven-parameters.

Hi @kimmte,
I've been unable to get this tested, so I'm very interested in hearing about your experience with these changes.

Thanks in advance
John

from extension-data-driven-parameters.

Previously mention alpha is now available as v2.0.0: https://extensiongallery.tableau.com/products/27.
Any updates on the performance issues when you tested locally?

Take care,
Keshia

from extension-data-driven-parameters.

Still unable to test locally. Despite commenting out the code locally, it still seems to run the original version. Removed the "mv" command from the package.json file (build script) before running "yarn build", since it caused errors on my machine, and tried copying the files directly from the build folder. Are you sure I'm not supposed to change anything else?

from extension-data-driven-parameters.

Hi @KeshiaRose , I’ve been able to comment the lines you mentioned above, but it seems like it’s still removing duplicates. For testing purposes, I inserted an alert line, and it works. I see the alert every time I make a selection. The sorting seems to be disable, which makes me think that there is a line somewhere that removes the duplicates, other than line 159.

Also – I followed the steps to deploy locally but the alert didn’t show up for me until I also modified the source location URL in the trex file to my localhost URL – can you please confirm that step is indeed needed?

Thanks,
Kim

from extension-data-driven-parameters.

Finally got the test working locally @KeshiaRose. Using alerts, I broke the routine down to several small components and there's only one that's taking any significant time and that's updating the state of the object at the very end of the routine:

        this.setState({
            currentVal,
            disabled: false,
            firstInit: false,
            list,
        });

Is there anything that can be done to improve this?

from extension-data-driven-parameters.

@kimmte: I can confirm that I did change the source location URL in the trex file to get the local copy working. I can also confirm that line 159 is the only thing in this extension that eliminates duplicates. Since I had created a sheet for the sole purpose of supporting this extension that only had one column on it, the sheet itself summarized away the duplicates that were in the raw data. By adding another column to the sheet I could see otherwise duplicate rows in the sheet and observe the Data Driven extension eliminating the duplicates as expected at line 159 only.

from extension-data-driven-parameters.

Hi @JohnDelahunt,

Thanks for testing out the duplication and running through the script! I was really hoping it would be one of the sorting or duplication pieces that were causing the performance issues as I'm not sure what I can do around setting the state to make it go faster. I'll look into any possible ways to speed this up.

Take care,
Keshi

from extension-data-driven-parameters.

Hi @kimmte,

Were you able to see improvements in performance without the sorting?

Take care,
Keshia

from extension-data-driven-parameters.

@kimmte Also, yes this is required:

Also – I followed the steps to deploy locally but the alert didn’t show up for me until I also modified the source location URL in the trex file to my localhost URL – can you please confirm that step is indeed needed?

from extension-data-driven-parameters.

@JohnDelahunt, is the delay happening on the Tableau parameter update? Would it help to just put the line (205) before setState, so the list (setState) can take its time to load but you don't wait on the actual Tableau parameter update?

from extension-data-driven-parameters.

@KeshiaRose No the delay is not in the parameter update. I split the parameter update out and almost all of the time passes in the setState step. (I say almost all of the time because the filter to eliminate duplicates takes about a second; everything else is practically instantaneous.) I understand that's more of a problem because React is not your code, but that's what I observed. I've been unable to figure out where the React library actually does anything for setState.

It baffles me that it takes four times as long each time the length of the list doubles. If all it's doing is copying it or moving it to another location then the relationship between list length and time should be linear in my mind. So it must be doing something more. If you can point me at the right part of the React library then maybe I can see how it uses the time.

A minor change you could make is to call setState before the list is built so that the end-user doesn't see "Parameter needs configuration" while the lengthy initialization takes place. How about "Loading..." instead? This is misleading for the end-user even if you are able to greatly shorten the time. No one needs end-users panicking over nothing.

from extension-data-driven-parameters.

@JohnDelahunt I can't seem to figure out how to make the setState step any faster and I'm not familiar with React enough to tell you where in the React library it executes it. However, I have added "Loading..." text while it is working so hopefully that will get rid of the confusion.

Take care,
Keshia

from extension-data-driven-parameters.

I appreciate that small change @KeshiaRose . Maybe someone else is following this thread who has more knowledge of React internals.

from extension-data-driven-parameters.

FYI @KeshiaRose I found this blog that takes my knowledge of setState to the next level (but not so far as to know what's making it slow in this case) https://overreacted.io/how-does-setstate-know-what-to-do/

from extension-data-driven-parameters.

I don't know what you did @KeshiaRose to make it happen, but a list of 10000 items now takes only 11 seconds to load and the full list of about 80000 items loads in under 3 minutes. That seems reasonable to me. Now it's up to us to convince our client to reduce the size of the list. Thanks.

from extension-data-driven-parameters.

Hah! That's awesome! I updated it this week to use the latest version of react-scripts which also meant upgrading react. Perhaps they enhanced the performance of setState! I'm going to close this but let me know if you need anything else. Super glad you were able to see improvements with the upgrade as I couldn't think of any more ideas on ways to speed it up.

On another note, for the love of god, yes, get them to decrease the number of items in the list! Actually, do you think an extension that allowed you to enter a search term, then fetched the list of matching values and allowed you to select from that to populate a parameter would be helpful for your use case? I can't imagine any end user wants to go through 80,000 items in a list. Let me know if you think there is a good use case there!

Take care,
Keshia

from extension-data-driven-parameters.