k0rventen / apple-health-grafana Goto Github PK
View Code? Open in Web Editor NEWVisualise your Apple Health export in Grafana
Visualise your Apple Health export in Grafana
When I tried to run this, I got this error:
ERROR: Service 'ingester' failed to build: When using COPY with more than one source file, the destination must be a directory and end with a /
I just added a "/" to the "." that was there in ingester/Dockerfile, and then it worked. So the new destination is: "./"
Ubuntu 20.04.6, Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1
ERROR: Service 'ingester' failed to build: When using COPY with more than one source file, the destination must be a directory and end with a /
version: "3"
services:
grafana:
image: grafana/grafana:10.0.3
ports:
- 3001:3001
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=health
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards/dashs/specific-metrics.json
volumes:
- ./provisioning:/etc/grafana/provisioning
influx:
image: influxdb:1.8
restart: always
environment:
- INFLUXDB_DB=health
volumes:
- influx-data:/var/lib/influxdb
ingester:
image: k0rventen/apple-health-grafana-ingester
build: ./ingester
volumes:
- ./export.zip:/export.zip
volumes:
influx-data:
Hi! First of all, thanks for the project, that's amazing! ❤️
I'm running into a crash on the ingester when importing my health data. Please find the logs below:
docker compose up ingester 1 ✘ 09:15:11
[+] Running 1/0
✔ Container apple-health-grafana-ingester-1 Created 0.0s
Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1 | Unzipping the export file...
apple-health-grafana-ingester-1 | Export file unzipped!
apple-health-grafana-ingester-1 | Influx is ready.
apple-health-grafana-ingester-1 | Loading workout routes ...
apple-health-grafana-ingester-1 | Opening Route 2022-03-07 2:14pm
apple-health-grafana-ingester-1 | Opening Route 2021-10-10 3:57pm
...
apple-health-grafana-ingester-1 | Opening Route 2022-11-18 8:07am
apple-health-grafana-ingester-1 | Opening Route 2021-04-20 7:14pm
apple-health-grafana-ingester-1 | Export file is /export/apple_health_export/export.xml
apple-health-grafana-ingester-1 | Traceback (most recent call last):
apple-health-grafana-ingester-1 | File "//app.py", line 185, in <module>
apple-health-grafana-ingester-1 | process_health_data(client)
apple-health-grafana-ingester-1 | File "//app.py", line 140, in process_health_data
apple-health-grafana-ingester-1 | for _, elem in etree.iterparse(export_file):
apple-health-grafana-ingester-1 | File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1249, in iterator
apple-health-grafana-ingester-1 | yield from pullparser.read_events()
apple-health-grafana-ingester-1 | File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1320, in read_events
apple-health-grafana-ingester-1 | raise event
apple-health-grafana-ingester-1 | File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1292, in feed
apple-health-grafana-ingester-1 | self._parser.feed(data)
apple-health-grafana-ingester-1 | xml.etree.ElementTree.ParseError: syntax error: line 156, column 0
apple-health-grafana-ingester-1 exited with code 1
Would you be able to take a look please?
It looks like "Apple Health" and "Apple health specific metrics" dashboards are not support data from "Shortcuts" app. Could it be fixed/improved?
Personally i have a lot data from shortcuts. I automated data import from external APIs to Apple Health, logging UV index when going outside, and simple shortcuts for logging body measurements.
Thank you!
Trying to ingest the data, the ingester container fails due to a ParseError
$ docker-compose up ingester
Creating applehealthgrafana_ingester_1 ... done
Attaching to applehealthgrafana_ingester_1
ingester_1 | unzipping the export file..
ingester_1 | export file unzipped
ingester_1 | influx is ready
ingester_1 | loading workout routes
ingester_1 | opening Route 2022-04-04 5:09pm
.....
ingester_1 | opening Route 2022-05-18 1:52pm
ingester_1 | Traceback (most recent call last):
ingester_1 | File "//app.py", line 147, in <module>
ingester_1 | process_health_data()
ingester_1 | File "//app.py", line 107, in process_health_data
ingester_1 | for _, elem in etree.iterparse(export_path):
ingester_1 | File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1249, in iterator
ingester_1 | yield from pullparser.read_events()
ingester_1 | File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1320, in read_events
ingester_1 | raise event
ingester_1 | File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1292, in feed
ingester_1 | self._parser.feed(data)
ingester_1 | xml.etree.ElementTree.ParseError: syntax error: line 156, column 0
applehealthgrafana_ingester_1 exited with code 1
The export.zip
contains the following files
$ ls -ltra
total 904176
-rw-r--r-- 1 root root 664488691 Mar 2 17:30 export.xml
-rw-r--r-- 1 root root 261333433 Mar 2 17:30 export_cda.xml
drwxr-xr-x 7 root root 4096 Mar 3 11:31 ..
drwxr-xr-x 2 root root 28672 Mar 3 11:32 workout-routes
drwxr-xr-x 2 root root 4096 Mar 3 11:32 electrocardiograms
drwxr-xr-x 4 root root 4096 Mar 3 11:32 .
Here is the container status:
#$ docker ps -a | grep health
693fa4d42d6f k0rventen/apple-health-grafana-ingester "/bin/sh -c 'pytho..." 58 minutes ago Exited (1) 55 minutes ago applehealthgrafana_ingester_1
6767e7e85b48 grafana/grafana:9.3.8 "/run.sh" 2 hours ago Up 2 hours 0.0.0.0:3000->3000/tcp applehealthgrafana_grafana_1
aae540aca1c6 influxdb:1.8 "/entrypoint.sh in..." 2 hours ago Up 2 hours 8086/tcp applehealthgrafana_influx_1
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services: 'influx'
At present only numeric data is ingested into the InfluxDB, but some metrics, for example AppleStandHour, SleepAnalysis uses a text value. Can these text based value be ingested into InfluxDB as well as the numeric data?
Hi, I'm trying to get my data in grafana, i found your importer which seems perfect but the import keeps crashing at Inserted 430000 records
over and over, always the same place, any ideas what might be going on here ?
export.zip is 131MB in size
Docker
health-ingester | Inserted 420000 records
influxdb | [httpd] 172.21.0.8 - root [23/Jun/2024:06:56:59 +0000] "POST /write?db=health&precision=s HTTP/1.1 " 204 0 "-" "python-requests/2.31.0" c995d268-312d-11ef-8ace-0242ac15001c 28687
health-ingester | Inserted 430000 records
health-ingester | Traceback (most recent call last):
health-ingester | File "//app.py", line 197, in <module>
health-ingester | process_health_data(client)
health-ingester | File "//app.py", line 146, in process_health_data
health-ingester | rec = format_record(elem)
health-ingester | ^^^^^^^^^^^^^^^^^^^
health-ingester | File "//app.py", line 64, in format_record
health-ingester | date = parse_date_as_timestamp(record.get("startDate", 0))
health-ingester | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
health-ingester | File "/formatters.py", line 19, in parse_date_as_timestamp
health-ingester | return int(dt.fromisoformat(v).timestamp())
health-ingester | ^^^^^^^^^^^^^^^^^^^
health-ingester | TypeError: fromisoformat: argument must be str
health-ingester exited with code 0
health-ingester | Unzipping the export file...
health-ingester | Export file unzipped!
influx:
image: influxdb:1.8
container_name: influxdb
restart: unless-stopped
environment:
- INFLUXDB_DB=health
volumes:
- ${CONFIG_ROOT}/influx:/var/lib/influxdb
ports:
- 8086:8086
logging:
driver: loki
options:
loki-url: ${LOKI_URL}
mode: non-blocking
max-buffer-size: 4m
loki-retries: "3"
ingester:
image: k0rventen/apple-health-grafana-ingester:v0.5.0
container_name: health-ingester
restart: unless-stopped
volumes:
- ${CONFIG_ROOT}/influx/export.zip:/export.zip
logging:
driver: loki
options:
loki-url: ${LOKI_URL}
mode: non-blocking
max-buffer-size: 4m
loki-retries: "3"
I just imported all my health data successfully. 😊
However, something seems to be not quite right with the numbers. 🤔
Here's a screenshot of the "steps per day" graph from the preset Grafana dashboard "Apple health specific metrics" where I added screenshots from the Fitness App that shows the steps data from three days.
The steps curve in the Grafana dashboard always shows much higher values. – Why?
after specifying the export.zip relative as proposed it fails with this error:
apple-health-grafana-ingester-1 | Unzipping the export file...
apple-health-grafana-ingester-1 | Export file unzipped!
apple-health-grafana-ingester-1 | Influx is ready.
apple-health-grafana-ingester-1 | No workout routes found, skipping ...
apple-health-grafana-ingester-1 | Traceback (most recent call last):
apple-health-grafana-ingester-1 | File "//app.py", line 186, in <module>
apple-health-grafana-ingester-1 | process_health_data(client)
apple-health-grafana-ingester-1 | File "//app.py", line 131, in process_health_data
apple-health-grafana-ingester-1 | export_xml_files = [f for f in os.listdir(EXPORT_PATH) if EXPORT_XML_REGEX.match(f)]
apple-health-grafana-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^
apple-health-grafana-ingester-1 | FileNotFoundError: [Errno 2] No such file or directory: '/export/apple_health_export'apple-health-grafana-ingester-1 exited with code 1
but ./export/apple_health_export/ exists,...
Get the following error even though export.zip is clearly a zip file.
[+] Running 1/0
✔ Container apple-health-grafana-ingester-1 Created 0.0s
Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1 | Unzipping the export file...
apple-health-grafana-ingester-1 | Unable to open export zip: /export.zip is not a zip file
apple-health-grafana-ingester-1 exited with code 1
Would it be possible to wrap it into Nextcloud application? Thank you for consideration.
Have a great day.
when using
docker-compose pull
I get the following
$ docker-compose pull
Pulling ingester (k0rventen/apple-health-grafana-ingester:latest)...
ERROR: missing signature key
Also, may or may not be related..
when using
docker-compose up ingester
Step 4/5 : COPY *.py .
ERROR: Service 'ingester' failed to build: When using COPY with more than one source file, the destination must be a directory and end with a /
OSX 10.13.6
n/a
using-the-template-data
doc up ingester fails with "Please make sure the libxml2 and libxslt development packages are installed."
What am I doing wrong?
Thx!
Bernd
Linux pi4 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
pi@pi4:/opt/containers/apple-health $ doc up ingester
Creating network "apple-health_default" with the default driver
Building ingester
Step 1/5 : FROM python:3.11-alpine
---> c33d907e4a1d
Step 2/5 : COPY requirements.txt .
---> Using cache
---> ea9eb21f3e79
Step 3/5 : RUN pip3 install -r requirements.txt
---> Running in 3888ff75b972
Collecting influxdb==5.3.1 (from -r requirements.txt (line 1))
Downloading influxdb-5.3.1-py2.py3-none-any.whl (77 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 2.5 MB/s eta 0:00:00
Collecting gpxpy==1.5.0 (from -r requirements.txt (line 2))
Downloading gpxpy-1.5.0.tar.gz (111 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.6/111.6 kB 5.9 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting lxml==4.9.3 (from -r requirements.txt (line 3))
Downloading lxml-4.9.3.tar.gz (3.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 6.4 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [3 lines of output]
Building lxml version 4.9.3.
Building without Cython.
Error: Please make sure the libxml2 and libxslt development packages are installed.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
ERROR: Service 'ingester' failed to build: The command '/bin/sh -c pip3 install -r requirements.txt' returned a non-zero code: 1
pi@pi4:/opt/containers/apple-health $ more docker-compose.yml
version: "3"
services:
grafana:
image: grafana/grafana:10.0.3
ports:
- 3000:3000
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=health
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards/dashs/sp
ecific-metrics.json
volumes:
- ./provisioning:/etc/grafana/provisioning
influx:
image: influxdb:1.8
restart: always
environment:
- INFLUXDB_DB=health
volumes:
- influx-data:/var/lib/influxdb
ingester:
image: k0rventen/apple-health-grafana-ingester
build: ./ingester
volumes:
- ./health-export.zip:/export.zip
volumes:
influx-data:
I'm on macOS 13.3.1 on a MacBook Pro 14" with M1 Pro Max CPU and 32GB RAM.
Everything looked good after I started the ingester.
However, after app. 4'30" the process exited with "code 137" after about 5 million records.:
apple-health-grafana-ingester-1 | inserted 5070000 records
apple-health-grafana-ingester-1 | inserted 5080000 records
apple-health-grafana-ingester-1 | inserted 5090000 records
apple-health-grafana-ingester-1 exited with code 137
No "failed" information of any kind was output. I guess that exiting with "code 137" is not good? Is it an error? Which one? What can I do?
Do i understand correctly that there is no sense in providing "All" in sleep sources as we have in "Apple health specific metrics"?
Originally posted by @MarshalX in #12 (comment)
first, thanks for doing a fix for the previous issue and for the cool tool!
So I removed everything and started from scratch again, according to the readme.
I have a new issue:
Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1 | Unzipping the export file...
apple-health-grafana-ingester-1 | Unable to open export zip: /export.zip is not a zip file
apple-health-grafana-ingester-1 exited with code 1
I tried different ways (one at a time) to define the location of the file:
- C:/temp/ap/apple-health-grafana:/export.zip
- C:\temp\ap\apple-health-grafana:/export.zip
- C:\\temp\\ap\\apple-health-grafana:/export.zip
I can confirm the export.zip is a zip file. I even unzipped and rezipped with 7zip as zip. Same result. But I guess it just can't find the path. So how do I need to specify it on Windows?
Thank you!
Not everyone has distances in km but use miles. The Grafana charts shows the unit in km even though the data is actually in miles. Can the unit be detected from the data and the chart display the correct unit?
my error is the following, what can i do to alleviate this issue?
Pulling ingester ... pulling from k0rventen/apple-heal...
ERROR: for ingester no matching manifest for linux/arm/v7 in the manifest list entries
ERROR: no matching manifest for linux/arm/v7 in the manifest list entries
Idea here is to automate the data updates. There are two possible workflows:
https://apps.apple.com/us/app/health-export-csv/id1477944755
https://apps.apple.com/us/app/simple-health-export-csv/id1535380115
https://apps.apple.com/us/app/health-auto-export-json-csv/id1115567069
https://apps.apple.com/us/app/health-auto-export-json-csv/id1115567069
More than happy to use my limited my skills to help but I'm not an expert and wanted to see if you were even interested.
This looks like an amazing app, but I am having trouble getting it started.
When I run the ingester I get:
No such file or directory: '/export/apple_health_export/workout-routes/'
❯ docker-compose up ingester
[+] Running 9/9
⠿ ingester Pulled 4.6s
⠿ 63b65145d645 Pull complete 0.8s
⠿ 781eddb6f342 Pull complete 1.0s
⠿ 1ae1f1ea756f Pull complete 1.3s
⠿ 5b3e7c82c61d Pull complete 1.4s
⠿ 59b6a1c4fe6c Pull complete 1.6s
⠿ 07c7907083b1 Pull complete 1.6s
⠿ 48a8ca346aeb Pull complete 2.6s
⠿ 99e3fc1df906 Pull complete 2.7s
[+] Running 1/1
⠿ Container apple-health-grafana-ingester-1 Created 0.1s
Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1 | unzipping the export file..
apple-health-grafana-ingester-1 | export file unzipped
apple-health-grafana-ingester-1 | influx is ready
apple-health-grafana-ingester-1 | loading workout routes
apple-health-grafana-ingester-1 | Traceback (most recent call last):
apple-health-grafana-ingester-1 | File "//app.py", line 140, in <module>
apple-health-grafana-ingester-1 | process_workout_routes()
apple-health-grafana-ingester-1 | File "//app.py", line 92, in process_workout_routes
apple-health-grafana-ingester-1 | for file in os.listdir(routes_path):
apple-health-grafana-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^
apple-health-grafana-ingester-1 | FileNotFoundError: [Errno 2] No such file or directory: '/export/apple_health_export/workout-routes/'
apple-health-grafana-ingester-1 exited with code 1
~/src/apple-health-grafana main ⇡ 6s
❯ docker-compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
apple-health-grafana-grafana-1 grafana/grafana "/run.sh" grafana 45 seconds ago Up 43 seconds 0.0.0.0:3000->3000/tcp
apple-health-grafana-influx-1 influxdb:1.8 "/entrypoint.sh infl…" influx 45 seconds ago Up 43 seconds 8086/tcp
There seems a problem finding the Export.xml file in the zip file if running on Docker Windows.
The filename in my export ist written with first letter upper case: Export.xml
apple-health-grafana-ingester-1 | opening Route 2022-07-21 9:11am
apple-health-grafana-ingester-1 | opening Route 2022-08-30 7:58am
apple-health-grafana-ingester-1 | opening Route 2022-08-08 9:30pm
apple-health-grafana-ingester-1 | opening Route 2022-08-01 8:20am
apple-health-grafana-ingester-1 | no export.xml file found, skipping
apple-health-grafana-ingester-1 | All done ! You can now check grafana.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.