k0rventen / apple-health-grafana Goto Github PK

View Code? Open in Web Editor NEW

328.0 7.0 24.0 2.6 MB

Visualise your Apple Health export in Grafana

Dockerfile 1.22% Python 88.98% Shell 9.80%

apple apple-health docker grafana influxdb python data-visualization health health-data

apple-health-grafana's People

Contributors

Stargazers

Watchers

apple-health-grafana's Issues

no export.xml file found, skipping

There seems a problem finding the Export.xml file in the zip file if running on Docker Windows.
The filename in my export ist written with first letter upper case: Export.xml


apple-health-grafana-ingester-1  | opening Route 2022-07-21 9:11am
apple-health-grafana-ingester-1  | opening Route 2022-08-30 7:58am
apple-health-grafana-ingester-1  | opening Route 2022-08-08 9:30pm
apple-health-grafana-ingester-1  | opening Route 2022-08-01 8:20am
apple-health-grafana-ingester-1  | no export.xml file found, skipping
apple-health-grafana-ingester-1  | All done ! You can now check grafana.

xml.etree.ElementTree.ParseError: syntax error on import

Hi! First of all, thanks for the project, that's amazing! ❤️

I'm running into a crash on the ingester when importing my health data. Please find the logs below:

docker compose up ingester                                                                                                                                               1 ✘  09:15:11
[+] Running 1/0
 ✔ Container apple-health-grafana-ingester-1  Created                                                                                                                                                                                         0.0s
Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1  | Unzipping the export file...
apple-health-grafana-ingester-1  | Export file unzipped!
apple-health-grafana-ingester-1  | Influx is ready.
apple-health-grafana-ingester-1  | Loading workout routes ...
apple-health-grafana-ingester-1  | Opening Route 2022-03-07 2:14pm
apple-health-grafana-ingester-1  | Opening Route 2021-10-10 3:57pm
...
apple-health-grafana-ingester-1  | Opening Route 2022-11-18 8:07am
apple-health-grafana-ingester-1  | Opening Route 2021-04-20 7:14pm
apple-health-grafana-ingester-1  | Export file is /export/apple_health_export/export.xml
apple-health-grafana-ingester-1  | Traceback (most recent call last):
apple-health-grafana-ingester-1  |   File "//app.py", line 185, in <module>
apple-health-grafana-ingester-1  |     process_health_data(client)
apple-health-grafana-ingester-1  |   File "//app.py", line 140, in process_health_data
apple-health-grafana-ingester-1  |     for _, elem in etree.iterparse(export_file):
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1249, in iterator
apple-health-grafana-ingester-1  |     yield from pullparser.read_events()
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1320, in read_events
apple-health-grafana-ingester-1  |     raise event
apple-health-grafana-ingester-1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1292, in feed
apple-health-grafana-ingester-1  |     self._parser.feed(data)
apple-health-grafana-ingester-1  | xml.etree.ElementTree.ParseError: syntax error: line 156, column 0
apple-health-grafana-ingester-1 exited with code 1

Would you be able to take a look please?

Shortcuts data source

It looks like "Apple Health" and "Apple health specific metrics" dashboards are not support data from "Shortcuts" app. Could it be fixed/improved?

Personally i have a lot data from shortcuts. I automated data import from external APIs to Apple Health, logging UV index when going outside, and simple shortcuts for logging body measurements.

Thank you!

At present only numeric data is ingested into the InfluxDB, but some metrics, for example AppleStandHour, SleepAnalysis uses a text value. Can these text based value be ingested into InfluxDB as well as the numeric data?

ingester container fails Importing data info influxdb

Trying to ingest the data, the ingester container fails due to a ParseError

$ docker-compose up ingester
Creating applehealthgrafana_ingester_1 ... done
Attaching to applehealthgrafana_ingester_1
ingester_1  | unzipping the export file..
ingester_1  | export file unzipped
ingester_1  | influx is ready
ingester_1  | loading workout routes
ingester_1  | opening Route 2022-04-04 5:09pm
.....
ingester_1  | opening Route 2022-05-18 1:52pm
ingester_1  | Traceback (most recent call last):
ingester_1  |   File "//app.py", line 147, in <module>
ingester_1  |     process_health_data()
ingester_1  |   File "//app.py", line 107, in process_health_data
ingester_1  |     for _, elem in etree.iterparse(export_path):
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1249, in iterator
ingester_1  |     yield from pullparser.read_events()
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1320, in read_events
ingester_1  |     raise event
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1292, in feed
ingester_1  |     self._parser.feed(data)
ingester_1  | xml.etree.ElementTree.ParseError: syntax error: line 156, column 0
applehealthgrafana_ingester_1 exited with code 1

The export.zip contains the following files

$ ls -ltra
total 904176
-rw-r--r-- 1 root root 664488691 Mar  2 17:30 export.xml
-rw-r--r-- 1 root root 261333433 Mar  2 17:30 export_cda.xml
drwxr-xr-x 7 root root      4096 Mar  3 11:31 ..
drwxr-xr-x 2 root root     28672 Mar  3 11:32 workout-routes
drwxr-xr-x 2 root root      4096 Mar  3 11:32 electrocardiograms
drwxr-xr-x 4 root root      4096 Mar  3 11:32 .

Here is the container status:

#$ docker ps -a | grep health
693fa4d42d6f        k0rventen/apple-health-grafana-ingester   "/bin/sh -c 'pytho..."   58 minutes ago      Exited (1) 55 minutes ago                                                                                applehealthgrafana_ingester_1
6767e7e85b48        grafana/grafana:9.3.8                     "/run.sh"                2 hours ago         Up 2 hours                  0.0.0.0:3000->3000/tcp                                                       applehealthgrafana_grafana_1
aae540aca1c6        influxdb:1.8                              "/entrypoint.sh in..."   2 hours ago         Up 2 hours                  8086/tcp                                                                     applehealthgrafana_influx_1

FileNotFoundError: [Errno 2] No such file or directory: '/export/apple_health_export'apple-health-grafana-ingester-1 exited

after specifying the export.zip relative as proposed it fails with this error:

apple-health-grafana-ingester-1  | Unzipping the export file...
apple-health-grafana-ingester-1  | Export file unzipped!
apple-health-grafana-ingester-1  | Influx is ready.
apple-health-grafana-ingester-1  | No workout routes found, skipping ...
apple-health-grafana-ingester-1  | Traceback (most recent call last):
apple-health-grafana-ingester-1  |   File "//app.py", line 186, in <module>
apple-health-grafana-ingester-1  |     process_health_data(client)
apple-health-grafana-ingester-1  |   File "//app.py", line 131, in process_health_data
apple-health-grafana-ingester-1  |     export_xml_files = [f for f in os.listdir(EXPORT_PATH) if EXPORT_XML_REGEX.match(f)]
apple-health-grafana-ingester-1  |                                    ^^^^^^^^^^^^^^^^^^^^^^^
apple-health-grafana-ingester-1  | FileNotFoundError: [Errno 2] No such file or directory: '/export/apple_health_export'apple-health-grafana-ingester-1 exited with code 1

but ./export/apple_health_export/ exists,...

Unable to open export zip: /export.zip is not a zip file

first, thanks for doing a fix for the previous issue and for the cool tool!

So I removed everything and started from scratch again, according to the readme.

I have a new issue:

Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1  | Unzipping the export file...
apple-health-grafana-ingester-1  | Unable to open export zip: /export.zip is not a zip file
apple-health-grafana-ingester-1 exited with code 1

I tried different ways (one at a time) to define the location of the file:

    - C:/temp/ap/apple-health-grafana:/export.zip
    - C:\temp\ap\apple-health-grafana:/export.zip
    - C:\\temp\\ap\\apple-health-grafana:/export.zip

I can confirm the export.zip is a zip file. I even unzipped and rezipped with 7zip as zip. Same result. But I guess it just can't find the path. So how do I need to specify it on Windows?

Thank you!

ERROR : missing signature key

What happened?

when using

docker-compose pull

I get the following

$ docker-compose pull
Pulling ingester (k0rventen/apple-health-grafana-ingester:latest)...
ERROR: missing signature key

Also, may or may not be related..

when using

docker-compose up ingester

Step 4/5 : COPY *.py .
ERROR: Service 'ingester' failed to build: When using COPY with more than one source file, the destination must be a directory and end with a /

Environment

OSX 10.13.6

Relevant log output

n/a

docker-compose.yml

using-the-template-data

No such file or directory: '/export/apple_health_export/workout-routes/'

This looks like an amazing app, but I am having trouble getting it started.

When I run the ingester I get:

No such file or directory: '/export/apple_health_export/workout-routes/'

❯ docker-compose up ingester
[+] Running 9/9
 ⠿ ingester Pulled                                                         4.6s
   ⠿ 63b65145d645 Pull complete                                            0.8s
   ⠿ 781eddb6f342 Pull complete                                            1.0s
   ⠿ 1ae1f1ea756f Pull complete                                            1.3s
   ⠿ 5b3e7c82c61d Pull complete                                            1.4s
   ⠿ 59b6a1c4fe6c Pull complete                                            1.6s
   ⠿ 07c7907083b1 Pull complete                                            1.6s
   ⠿ 48a8ca346aeb Pull complete                                            2.6s
   ⠿ 99e3fc1df906 Pull complete                                            2.7s
[+] Running 1/1
 ⠿ Container apple-health-grafana-ingester-1  Created                      0.1s
Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1  | unzipping the export file..
apple-health-grafana-ingester-1  | export file unzipped
apple-health-grafana-ingester-1  | influx is ready
apple-health-grafana-ingester-1  | loading workout routes
apple-health-grafana-ingester-1  | Traceback (most recent call last):
apple-health-grafana-ingester-1  |   File "//app.py", line 140, in <module>
apple-health-grafana-ingester-1  |     process_workout_routes()
apple-health-grafana-ingester-1  |   File "//app.py", line 92, in process_workout_routes
apple-health-grafana-ingester-1  |     for file in os.listdir(routes_path):
apple-health-grafana-ingester-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^
apple-health-grafana-ingester-1  | FileNotFoundError: [Errno 2] No such file or directory: '/export/apple_health_export/workout-routes/'
apple-health-grafana-ingester-1 exited with code 1

~/src/apple-health-grafana main ⇡ 6s
❯ docker-compose ps
NAME                             IMAGE               COMMAND                  SERVICE             CREATED             STATUS              PORTS
apple-health-grafana-grafana-1   grafana/grafana     "/run.sh"                grafana             45 seconds ago      Up 43 seconds       0.0.0.0:3000->3000/tcp
apple-health-grafana-influx-1    influxdb:1.8        "/entrypoint.sh infl…"   influx              45 seconds ago      Up 43 seconds       8086/tcp

Unable to open export zip: /export.zip is not a zip file

Get the following error even though export.zip is clearly a zip file.

[+] Running 1/0
✔ Container apple-health-grafana-ingester-1 Created 0.0s
Attaching to apple-health-grafana-ingester-1
apple-health-grafana-ingester-1 | Unzipping the export file...
apple-health-grafana-ingester-1 | Unable to open export zip: /export.zip is not a zip file
apple-health-grafana-ingester-1 exited with code 1

Trying to install on raspberry pi 4 but get an error when using docker-compose pull

my error is the following, what can i do to alleviate this issue?

Pulling ingester ... pulling from k0rventen/apple-heal...

ERROR: for ingester no matching manifest for linux/arm/v7 in the manifest list entries
ERROR: no matching manifest for linux/arm/v7 in the manifest list entries

Assumption that Distances are km

Not everyone has distances in km but use miles. The Grafana charts shows the unit in km even though the data is actually in miles. Can the unit be detected from the data and the chart display the correct unit?

`database name required`

Sleep data sources

Do i understand correctly that there is no sense in providing "All" in sleep sources as we have in "Apple health specific metrics"?

Originally posted by @MarshalX in #12 (comment)

[Bug]: doc up ingester fails with "Please make sure the libxml2 and libxslt development packages are installed."

What happened?

doc up ingester fails with "Please make sure the libxml2 and libxslt development packages are installed."

What am I doing wrong?

Thx!
Bernd

Environment

Linux pi4 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux

Relevant log output

pi@pi4:/opt/containers/apple-health $ doc up ingester
Creating network "apple-health_default" with the default driver
Building ingester
Step 1/5 : FROM python:3.11-alpine
 ---> c33d907e4a1d
Step 2/5 : COPY requirements.txt .
 ---> Using cache
 ---> ea9eb21f3e79
Step 3/5 : RUN pip3 install -r requirements.txt
 ---> Running in 3888ff75b972
Collecting influxdb==5.3.1 (from -r requirements.txt (line 1))
  Downloading influxdb-5.3.1-py2.py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 2.5 MB/s eta 0:00:00
Collecting gpxpy==1.5.0 (from -r requirements.txt (line 2))
  Downloading gpxpy-1.5.0.tar.gz (111 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.6/111.6 kB 5.9 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting lxml==4.9.3 (from -r requirements.txt (line 3))
  Downloading lxml-4.9.3.tar.gz (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 6.4 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Building lxml version 4.9.3.
      Building without Cython.
      Error: Please make sure the libxml2 and libxslt development packages are installed.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
ERROR: Service 'ingester' failed to build: The command '/bin/sh -c pip3 install -r requirements.txt' returned a non-zero code: 1

docker-compose.yml

pi@pi4:/opt/containers/apple-health $ more docker-compose.yml
version: "3"
services:
  grafana:
    image: grafana/grafana:10.0.3
    ports:
    - 3000:3000
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=health
      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards/dashs/sp
ecific-metrics.json
    volumes:
    - ./provisioning:/etc/grafana/provisioning
  influx:
    image: influxdb:1.8
    restart: always
    environment:
      - INFLUXDB_DB=health
    volumes:
    - influx-data:/var/lib/influxdb
  ingester:
    image: k0rventen/apple-health-grafana-ingester
    build: ./ingester
    volumes:
    - ./health-export.zip:/export.zip

volumes:
  influx-data:

[Feature Request] - Integration with Auto Extractors

Idea here is to automate the data updates. There are two possible workflows:

https://apps.apple.com/us/app/health-export-csv/id1477944755
https://apps.apple.com/us/app/simple-health-export-csv/id1535380115
https://apps.apple.com/us/app/health-auto-export-json-csv/id1115567069

This application would autoextract the data to an icloud location
User would be responsible for getting the data to the import directory. There are other project that can do this.
This application would scan that directory on update, accept the format of the export and import the new data.

https://apps.apple.com/us/app/health-auto-export-json-csv/id1115567069

This workflow would have the data call out to an endpoint, rest or MQTT
MQTT woudl probably be the easiest to integrate
Here is an http endpoint to receive the rest data
https://github.com/irvinlim/apple-health-ingester
This application would listen on an mqtt broker and update the influx db upon publish events.

More than happy to use my limited my skills to help but I'm not an expert and wanted to see if you were even interested.

Numbers don't match. Example: steps count per day

I just imported all my health data successfully. 😊

However, something seems to be not quite right with the numbers. 🤔

Here's a screenshot of the "steps per day" graph from the preset Grafana dashboard "Apple health specific metrics" where I added screenshots from the Fitness App that shows the steps data from three days.

The steps curve in the Grafana dashboard always shows much higher values. – Why?

Unsupported config option for services: 'influx'

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services: 'influx'

ingester exits with "code 137" while importing 😞

I'm on macOS 13.3.1 on a MacBook Pro 14" with M1 Pro Max CPU and 32GB RAM.
Everything looked good after I started the ingester.

However, after app. 4'30" the process exited with "code 137" after about 5 million records.:

apple-health-grafana-ingester-1  | inserted 5070000 records
apple-health-grafana-ingester-1  | inserted 5080000 records
apple-health-grafana-ingester-1  | inserted 5090000 records
apple-health-grafana-ingester-1 exited with code 137

No "failed" information of any kind was output. I guess that exiting with "code 137" is not good? Is it an error? Which one? What can I do?

[Bug]: docker compose up ingester COPY error

What happened?

When I tried to run this, I got this error:
ERROR: Service 'ingester' failed to build: When using COPY with more than one source file, the destination must be a directory and end with a /

I just added a "/" to the "." that was there in ingester/Dockerfile, and then it worked. So the new destination is: "./"

Environment

Ubuntu 20.04.6, Docker version 24.0.5, build 24.0.5-0ubuntu1~20.04.1

Relevant log output

ERROR: Service 'ingester' failed to build: When using COPY with more than one source file, the destination must be a directory and end with a /

docker-compose.yml

version: "3"
services:
  grafana:
    image: grafana/grafana:10.0.3
    ports:
    - 3001:3001
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=health
      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards/dashs/specific-metrics.json
    volumes:
    - ./provisioning:/etc/grafana/provisioning
  influx:
    image: influxdb:1.8
    restart: always
    environment:
      - INFLUXDB_DB=health
    volumes:
    - influx-data:/var/lib/influxdb
  ingester:
    image: k0rventen/apple-health-grafana-ingester
    build: ./ingester
    volumes:
        - ./export.zip:/export.zip

volumes:
  influx-data:

k0rventen / apple-health-grafana Goto Github PK

apple-health-grafana's People

Contributors

Stargazers

Watchers

Forkers

apple-health-grafana's Issues

What happened?

Environment

Relevant log output

docker-compose.yml

What happened?

Environment

Relevant log output

docker-compose.yml

What happened?

Environment

Relevant log output

docker-compose.yml

Recommend Projects

Recommend Topics

Recommend Org