Getting Started

Getting Started#

Smart Health#

Overview#

Smart Health is a research initiative collecting smartphone and wearable data from up to 10,000 consenting participants over five years (sponsored by the University of Sheffield). This project aims to link this data to NHS records via the local Subnational Data Environment for Yorkshire and Humber.

This repository provides the pipeline to process raw data received from Daiser and build a structured database for future Smart Health research projects, ensuring data from various devices is stored securely and is accessible for linkage and analysis.

Setup#

Prerequisites#

  • uv: The Python package installer. See the official installation guide for setup.

  • Git: For version control.

  • 7zip file archiver. Ensure not to use the obsolete p7zip port. On Linux, run sudo apt install 7zip.

Installation Steps#

  1. Clone the Repository:

    git clone git@github.com:Data-Connect-UoS/BHF-SmartHealth.git 
cd BHF-SmartHealth

  2. Install Dependencies: Use uv sync to install the project’s core runtime dependencies.

  3. Install pre-commit Hooks: Run uv run pre-commit install to install the project’s pre-commits hooks.

Running Data Pipelines (Snakemake)#

This project uses Snakemake to execute the data processing pipelines.

Prerequisites#

  • Ensure you have installed all dependencies using uv sync.

  • You must create a configuration file named secrets.yaml in the workflow/config/ directory. This file must contain the necessary URLs and keys for accessing the S3 data buckets. An example structure is provided in workflow/config/example_secrets.yaml.

Available Pipelines#

The following Snakemake workflow files are available within the workflow/ directory:

Pipeline File

Description

Output

workflow/Snakefile_Device.smk

Processes raw Fitbit device data (daily summary and intraday).

Summarised binary files

workflow/Snakefile_GPS.smk

Processes raw GPS data.

Summarised LSOA and Mobility binary files

Dry Run (Testing)#

Perform a dry run to test the pipeline logic, check file dependencies, and identify potential errors without executing any steps:

uv run snakemake -s workflow/<PipelineFile> --dryrun

Execution#

To execute a pipeline, use the –cores flag to specify parallelism. If no value is specified, Snakemake will automatically use the number of available CPU cores.

uv run snakemake -s workflow/<PipelineFile> --cores

Contributing and Developer Guides#

The project documentation is built using Sphinx, with API reference documentation automatically generated from Python docstrings using sphinx-autoapi.

A CI/CD workflow automatically builds and deploys the documentation in main to GitHub Pages.

For all development standards, including:

  • Detailed local documentation build steps

  • Coding standards (Ruff, formatting)

  • The full list of automated CI/CD checks (linting, testing, cleansing)

Please consult the dedicated CONTRIBUTING.md document.

On this page