# Raw Data Description ## Data Streams from Daiser This data will be ingressed via a company [Daiser](https://daiser.com/). Daiser are building an app that will collate data from participants' wearable devices on their smart phones. This data will be provided to Data Connect by ingress to our RONIN Isolate Secure Data Environment (SDE). The Daiser app will receive data from the following streams: * Fitbit API * Health app data (i.e. data collected from other wearable devices through Apple Health or Google Fit) * Smartphone GPS data * Follow-up health questionnaires ```{figure} ../_static/daiser_streams.webp :align: center :width: 700px :alt: The user installs the Daiser app on their smartphone which syncs with Fitbit servers. The Daiser app combines data from the Fitbit API with smartphone GPS and Daiser Health Kit data, before sending a nightly encrypted and compressed data file to the University of Sheffield Trusted Research Environment (TRE). Daiser data flow ``` The inital MVP phase of the pipeline build includes processing Fitbit and GPS data recieved from Daiser. --- ## Incoming Data Structure Daily data will be received in a 7z[^sn1] archive. Within this archive there will be 8 directories: One for each of the previous 7 days (with the date as the directory name, in the format YYYMMDD), and one for the follow-up questionnaire data. [^sn1]: 7z compression is a high-efficiency compression method primarily associated with open-source software [7-Zip](https://www.7-zip.org/7z.html). The internal structure of each of the daily data directories will be identical. Each will contain a “GPS” and “Device” subdirectory, and within those there will be a directory for each individual participant. Within each participant in the “Device” directory structure, there will always be a Health Application directory (this is the data collected through the Daiser Smart Health mobile app) containing the daily summary data for their wearable devices. If the participant has a Fitbit device and has consented to sharing that data then there will also be a Fitbit directory containing the daily summary data and the intraday data directory from their Fitbit device. The diagram below shows the internal directory structure for data processed by the pipeline developed within the MVP. ```{figure} ../_static/incoming_data.webp :align: centre :alt: The daily Daiser Archive is a single 7Zip archive containing the previous 7 days' worth of data in seperate sub-directories. Each dated subdirectory will contain a GPS and a Device subdirectory. These subdirectories will contain another subdirectory per participant. For the GPS data, each participant subdirectory will contain one .json file containing GPS data. For the Device data, each participant will have a Fitbit subdirectory containing three json files; one containing Daily Summary data, one containing intraday heart rate data, and one containing steps data Diagram of the directory structure within the compressed daily archive recieved from Daiser. Note that this diagram represents the data processed by the pipeline developed in the MVP. ``` #### Example folder structure Below is an example of the file structure and naming within the compressed daily archive. ```{code-block} shell ├── DaiserArchive_20251030​​/ | ├── 20251023/ | | ├── Device/ | | | ├── UOS001/Fitbit/ | | | | ├── UOS001_20251023_heart.json | | | | ├── UOS001_20251023_fitbit.json | | | ├── UOS001/healthapp/ | | | | ├── UOS001_20251023_healthdata.json | | ├── GPS | | | ├── UOS001/UOS001_20241023_gps.json | ├── 20251024/ | | ├── Device/ | | | ├── UOS001/ | | | | ├── UOS001_20251024_heart.json | | | | ├── UOS001_20251024_fitbit.json | | ├── GPS | | | ├── UOS001/UOS001_20241024_gps.json | ├── 20251025/ | | ├── Device/ | | | ├── UOS001/ | | | | ├── UOS001_20251025_heart.json | | | | ├── UOS001_20251025_fitbit.json | | ├── GPS | | | ├── UOS001/UOS001_20251025_gps.json | ├── 20251026/ | | ├── Device/ | | | ├── UOS001/ | | | | ├── UOS001_20251026_heart.json | | | | ├── UOS001_20251026_fitbit.json | | ├── GPS | | | ├── UOS001/UOS001_20251026_gps.json ....same for 20251027,20251028 and 20251029 ``` #### Example JSON structures This section contains more information about the structure of the `.json` files expected for each data stream. Each data stream has a number of *required* data fields. The Fitbit daily summary file may also contain a number of optional fields. More information is provided below. Where a participant's file contains data fields that were not expected, a warning is raised, the file __is not processed__, and the pipeline continues processing the next file. _Note that it is possible that a user does not sync their Fitbit device on any given day. The Intraday data received for these days will be empty json arrays. The Daily Summary data would only include the fields synthesised by Fitbit (rather than measured by the device). These fields are `calories` (the users' basal metabolic rate with no measured additional calories) and `minutesSedentary` (caculated as the number of minutes in the day that the user was not active above sedentary level)._ ##### Fitbit Daily Summary The daily summary data file contains a number of daily activity summary metrics. Some of these metric fields are mandatory, whilst others are optional (meaning a participant's daily data file may not contain this data). The mandatory fields for the Daily Summary data are: * `calories` * `steps` * `minutesSedentary` * `minutesLightlyActive` * `minutesFairlyActive` * `minutesVeryActive` * `HeartRateCardioMinutes` * `HeartRateFatBurnMinutes` * `HeartRateResting` The optional Daily Summary data fields are: * `caloriesBMR` * `distance` * `elevation` * `floors` * `breathingRate` * `vo2Max` * `dailyRmssd` * `deepRmssd` * `sleep_deep_minutes` * `sleep_light_minutes` * `sleep_rem_minutes` * `sleep_wake_minutes` * `sleep_minutesToFallAsleep` * `sleep_timeInBed` * `SP02_avg` * `SP02_min` * `SP02_max` * `tempCore` The full list of possible metrics and their units can be found in the [data dictionary](../database_design/database_design.md#data-dictionary-mvp-only). Each summary metric object will contain an array of a single object containing the necessary `dateTime`, `value`, and `units` fields of that particular metric, as in the example below: ```{code-block} json { "calories": [ { "dateTime": "2018-12-26", "value": "11" "units": "calories" } ], ..., } ``` An example Fitbit Daily Summary `.json` file can be found [on the source repository](https://github.com/Data-Connect-UoS/BHF-SmartHealth/blob/main/tests/data/user01_2018-12-26.json). ##### Fitbit Intraday Activities Each activity stream, e.g. `heart` or `steps`, will be recieved in a separate file. This file will have a nested structure. An `activities--intraday` object which contains a `dataset` array of the individual date-value objects. These individual objects must contain a `d` (datetime) and `v` (value). See the example below. ```{code-block} json { "activities-steps-intraday": { "dataset": [ { "d": "2025-07-14T00:00:05Z", "v": 0 }, { "d": "2025-07-14T00:01:11Z", "v": 0 }, ... ] } ..., } ``` An example Fitbit intraday `.json` file containing steps data can be found [on the source repository](https://github.com/Data-Connect-UoS/BHF-SmartHealth/blob/main/tests/data/user01_steps.json). ##### GPS data The GPS data file has a nested object structure. A `gps-coordinates` object contains a `dataset` array containing the individual value objects. Each individual value contains the required fields `d` (datetime), `lat` (latitude) and `long` (longitude), as in the example below: ```{code-block} json "gps-coordinates": { "dataset": [ { "d": "2025-10-09T00:00:00", "lat": 53.3806626, "long": -1.4702278 }, ] } ``` An example GPS data `.json` file can be found [on the source repository](https://github.com/Data-Connect-UoS/BHF-SmartHealth/blob/main/tests/data/synth_gps_2025-10-09.json). ##### Health App and Baseline/ Follow-up Questionnaire data These datasets are outside of the scope of the initial MVP phase.