Comments (6)
Sweet, your documentation fixes in the README set me straight. That looks like it worked! Off to go try out playing with the configuration and loading some data in! Thank you!
$ curl -X POST "http://localhost:8989/api/data/insert/events?api_key=local" --data '{"user": "alice", "event": "click"}'
ok
$ curl -G "http://localhost:8989/api/data/query" \ \
--data-urlencode "api_key=local" \
--data-urlencode "query=select * from events"
[
{"__row_id":1778898631140184064,"event":"click","user":"alice"}
]
from scratchdata.
For the hobbyist interested in trying out this tool, what config options would I need to set to either meet the authentication requirements, or disable them, or run the program in a different way in order to be able to try out this tool?
In terms of my skill level, I write SQL all day, but I've never managed to set up an HTTPS server -- only http.
from scratchdata.
Hey @NortySpock, thanks for checking out this project! This has been something we have been meaning to fix (hence why the issue exists). I can talk with @poundifdef tomorrow about how we want to fix this and get a fix up.
from scratchdata.
If you just run go run .
then you should be able to follow the steps in the readme and most everything should work. The API portion of the project will work using a default local config on a local DuckDB instance. If you modify config.yaml
(to connect to an external database) then that'll work as well. It does not use HTTPS locally.
Thing that does not, work locally is the UI because currently requires google auth. @breadchris is generously working on a fix for this, so stay tuned!
Can you share more about what you're looking to do? Happy to give more specific help too.
from scratchdata.
If you just run
go run .
then you should be able to follow the steps in the readme and most everything should work.
I had had another process using port 8080, so I had wanted to use a different port (hence my starting with the config file rather than using the defaults). Turning that process off and running ./scratchdata
, it spins up just fine.
Running along with your quickstart, curl -X POST "http://localhost:8080/api/data/insert/events&api_key=local" --data '{"user": "alice", "event": "click"}'
returns
Unauthorized
to curl, with the logs showing
2024/04/12 07:42:55 /home/runner/work/scratchdata/scratchdata/pkg/storage/database/gorm/gorm.go:304 record not found
[1.451ms] [rows:0] SELECT * FROM `api_keys` WHERE hashed_api_key = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" AND `api_keys`.`deleted_at` IS NULL ORDER BY `api_keys`.`id` LIMIT 1
Possibly I am using a slightly different version of curl? I tried --json but that was an unrecognized command line option. I'm sure I'm doing something wrong, and I have not had a lot of spare time to troubleshoot :)
$ curl --version
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.17
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd
Can you share more about what you're looking to do? Happy to give more specific help too.
Just some simple data exploration -- home server metrics, log entries, garbage collection cycle statistics, some smart-home events, maybe go on a novice spelunking tour of some stock market data. I'm running pretty low-end hardware at home, and I became infatuated with the Benthos and VictoriaMetrics tools, which make it a snap to stream data in via just converting the data to json and firing it at an API endpoint. But VictoriaMetrics is not a general purpose columnar database, it's tightly focused around metrics. Scratchdata + DuckDB sounded like a really nice pairing to let me automate the data ingestion without having to think about table schema much ahead of time. Once I get this working, I was going to use dbt to create views to slice, dice, and remix the data. It's just that SQLite is a little slower at bulk analytics than I have patience for :)
Edit: just re-tested again, having re-downloaded the release for v1.0.14 ... same error. Tried deleting the data folder and starting "from nothing", same error.
Edit2: the storage/database/gorm/gorm.go:304 record not found
error makes me think perhaps scratchdata is trying to fetch a record out of the database... If the database had just been created from a first-time start, it might have nothing in that table...
from scratchdata.
I figured it out. There is a typo in our curl command in the docs which I'm fixing now. To insert data, you want to use ?
instead of &
for the query separator.
-/api/data/insert/events&api_key=local"
+/api/data/insert/events?api_key=local"
For changing the port: you can run ./scratchdata your_custom_config.yaml
. So if you copy the default config from the repo, you can change the port from 8080 to something else in the api.port
setting.
I also see that the --json
parameter was only added to curl in 2022, so that may not be present everywhere. Using --data
should work and I'll update the docs there as well.
from scratchdata.
Related Issues (20)
- Redshift Destination HOT 8
- Databricks destination HOT 1
- Use a AWS Secret Manager for DB credentials HOT 9
- Coerce BigQuery types when inserting HOT 5
- Check for available disk space
- Pause and resume data ingest
- Global lock
- move frontend into repo HOT 1
- implement postgres queue HOT 1
- duckdb integration
- tables need to be namespaced HOT 2
- credential share link
- break out router.go
- support more destinations
- Share data
- Cache API keys
- Queries as an API
- API key manager
- UI for Database Connection Add and Delete HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scratchdata.