Comments (23)
I was wondering if would not be better change the approach.
The main objective is to have the queries versioned, right?
We know that Git is a great versioning system. We could use it.
In my conception we can use GitHub as a "queries source".
Basically it would work as follows: In Redash settings we would have to set up:
- REDASH_GITHUB_REPO
- REDASH_GITHUB_USERNAME
- REDASH_GITHUB_ACCESS_TOKEN
In the UI, on the query creation page, we would have a field where we would put the path to the query
For example: path/to/query.sql
.
Our handler will hit the GitHub servers at https://api.github.com/repos/{repo}/contents/{path_to_query}
, given the auth previously get.
I even started to make a prototype, but not finished it.
If someone find it is worth, please feel free to fork me.
I think we could save a good development time with this idea and get the queries versioning working 😄
from redash.
any news about this features ?
from redash.
Along the same lines, comparing results between queries could be very useful too. A "Compared to previous version" tab next to the "Results" tab would be neat-o. I often export results to CSV files and compare them with daff.
from redash.
👍 This would be great, and the lack of it was one of the key cons on my list of deciding whether or not to adopt redash.
from redash.
+1
from redash.
Daff has a JS library, so it could perhaps even be integrated directly into the frontend.
from redash.
@whummer has a PR #1113 to implement a changes table. This will be storing the versions of the query but not currently presenting a UI to compare versions.
from redash.
@tsibley I'm planning to give access to past query results (they are stored anyway). Will keep daff in mind when it happens.
from redash.
@WesleyBatista It's a good idea, and one I've sketched out on paper for working with a local clone with bidirectional sync rather than assuming Github. I definitely like the idea of being able to look at query history in Git with my normal tools.
At the same time, such a change would require a substantial rework of the current backing datastore for queries and query-related data, and this might be prohibitive to a short-term fix. There's also nothing in particular that precludes doing a pluggable, Git-based query store later even if the project implements a non-Git query versioning scheme first for the short term.
from redash.
Using Git definitely crossed my mind and I even wrote a script that exports all queries as files to a folder that is managed with Git.
But the challenge for this feature is not only with how to store queries but also in building the appropriate UI, which using Git doesn't solve. I think the right way to go will be to have an optional Git store, but not mandatory.
from redash.
Totally agree with you guys! This can not be considered as a final solution. Surely it is an idea to be worked.
I just guessed that worth raise the thoughts.
The idea is to provide an optinal query source. We would still use the queries stored in-house.
But I was wondering about the requirements to get the idea working. What I thought til now was:
- a "query url" text field on the query source page.
- store the query url in the query record.
We can hide the feature on the early steps requiring to set REDASH_ENABLE_EXTERNAL_QUERY_SOURCE to True on the environment.
To store the "query url" we can create a migration to add a 'options' column on queries
table like we have on data_sources
.
As next steps we could think in improve the UI. The final one could be something more like a modal, navigating through the directories retrieved from the repo. Maybe add support to gists too... I dont know, just wondering.
So the tech debits that I found:
- queries stored outside redash will not be able to be scheduled unless we change some points on the scheduler (redash.tasks.execute_query, redash.tasks.QueryExecutor and redash.tasks.enqueue_query) to accept
- cache the queries (regarding the scheduler). request only if it changed on the source.
- improve the UI
Please, sorry if I am being too ingenuous and missed something (I probably did it). What do you guys think?
from redash.
And what do you think about the queries.options
column? I think that it actually would be like a "tech credit" 😜
With a JSON serializable available on queries record level we can open doors to get other features/ideas.
from redash.
@WesleyBatista I think an external query source likely wants to be more general than "fetch from a URL". Just like there are different data sources, there should probably be different query providers you can plug in and use. One provider might fetch from a URL, one from git, and another from a database itself. One way to design it might be to have the default query provider be a database provider pointed at re:dash's own database.
I expect, as you identified as a possibility, that any solution would want to cache query sources somewhere and have a means for updating/invalidating that cache. The benefit of using the current queries
table as the local cache is that other parts of redash can remain the same which simplifies the overall changes necessary.
A design that's appealing to me (and common in many areas) is something like a cascading provider list where, when it comes time to update a query source, each provider is tried in turn to provide the query source until one is able to. When it is, the query source is cached and the cache provider will be able to return it sooner the next time as long as it hasn't expired.
from redash.
hello ... I would give the redash project development
especially ui of it ...
help me for optimized architecture?
i want create a folder beside "app" in client => client/myProject
from redash.
Any new about the? It would be a great option to have and would make Redash so compelling.
Also instead of limiting this to just Github, it would be preferable to open this to any generic Git repo server so that any self-hosted, or SAS services Git servers out there can be used (over SSH)-- and even allow a localhost repos on a Redash instance...
from redash.
Any updates here.
from redash.
Nothing to share at this time. Later this year we plan to have a more transparent roadmap / engagement with contributors. We'll announce that on the user forum.
from redash.
Any update here?
from redash.
Not at this time.
from redash.
I saw that mozilla#149 was merged, that adds query versioning, or? If so which version of redash will this be included in? I don't see it in the version we are using.
from redash.
Mozilla haven't upstreamed this to the primary redash repository so there's no roadmap to include their change here.
from redash.
Are the query versions already stored in the database so I can access them manually?
from redash.
Are there any more updates on this? This would be a very useful feature to have.
from redash.
Related Issues (20)
- cannot import name 'DEPRECATED_SERVICE_NAMES' from 'botocore.docs' while using google sheets HOT 1
- Unsupported OP_QUERY command: connectionStatus. The client driver may require an upgrade HOT 3
- Dashboards doesn't work after upgrade to 10 version. HOT 6
- Mongo Connector Fails in loading any data due to unserializable Objectid property HOT 3
- Redash WebApp and Redash API are giving different results for the same query HOT 2
- Mongo Query Runner Processing time is broken HOT 1
- Add UI Warning/Error for Queries which have Alerts configured
- Refresh Schedule not working HOT 1
- Errors of queries will not show from ClickHouse v24 and v23 HOT 3
- postgres data source with 2 databases cannot connect to other databases "SSL connection has been closed unexpectedly"
- A db migration does not take Query#apply_auto_limit in count HOT 4
- The query result displays large numbers as NaN HOT 1
- Can not upgrade redash/base to redash/preview HOT 14
- Error running query: 'ascii' codec can't encode characters in position HOT 3
- facing internal server error after clicking on queries option HOT 2
- Add generic OAuth2/OIDC authentication HOT 1
- Uncaught promise error in Edit Visualization Dialog Modal
- date parameter configuration in redash self hosting. HOT 3
- Redash does not connect to Microsoft SQL Azure HOT 2
- queury in queue issue
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redash.