Process raw Fitbit Daily Summary data

Process raw Fitbit Daily Summary data#

This notebook demonstrates the use of Smart Health modules to process synthetic raw Fitbit Daily Summary data ready for ingress to the “Intermediate” bucket.

# Import required libraries and modules
import logging
from pathlib import Path
from bhf_smarthealth.parse import summary_parser as bhf_sum

# Initiate logging
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

# Path to raw example data
data_path = Path("../../../tests/data/raw/UOS001_20181226_fitbit.json")

  1. The raw data is loaded as a dictionary object

data_dict = bhf_sum.load_json(data_path)
data_dict

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[2], line 1
----> 1 data_dict = bhf_sum.load_json(data_path)
      2 data_dict

File ~/work/BHF-SmartHealth/BHF-SmartHealth/src/bhf_smarthealth/parse/summary_parser.py:43, in load_json(json_path)
     17 """
     18 Load the raw JSON file, reshape it into a flat dictionary, and validate
     19 data consistency against filename metadata.
   (...)     40     - If the filename date is unknown (Mandatory Error).
     41 """
     42 # Load raw JSON data
---> 43 json_dict = data_io.load_data_from_file(json_path)
     45 if not json_dict:
     46     raise EmptyFileError(f"No data to process in file {json_path}")

File ~/work/BHF-SmartHealth/BHF-SmartHealth/src/bhf_smarthealth/utils/data_io.py:48, in load_data_from_file(file_path, **kwargs)
     45     case _:
     46         raise ValueError("File extension not recognised.")
---> 48 with open(
     49     file_path,
     50     mode,
     51 ) as file:
     52     data = load_func(file, **kwargs)
     54 logger.debug(f"Data successfully loaded from {file_path}")

FileNotFoundError: [Errno 2] No such file or directory: '../../../tests/data/raw/UOS001_20181226_fitbit.json'

  1. The raw data dictionary is validated

This step checks that:

  • Mandatory fields are present

  • Which optional fields are present (including logging this)

  • That no unexpected fields are present - if they are, the file will be rejected by the pipeline

No ValidationError indicates that the step has run successfully

parsed_data = bhf_sum.validate_data(data_dict)

  1. A Pandas DataFrame is created from the parsed data

This DataFrame is now ready for combination with any Daily Summary metrics derived from the Intraday measures, such as daily average Heart Rate

There is only one row as each Daily Summary file contains one user’s Daily Summary data for a single day

The DataFrames for all users’ Daily Summary data processed on a given day will then be combined into one DataFrame before joining to the derived Daily metrics

data_df = bhf_sum.create_dataframe(parsed_data)

# Show the dataframe
data_df