bhf_smarthealth.parse.qualtrics_parser

This module contains functions to process raw Qualtrics data ingressed to the Smart Health pipeline

Attributes

logger

Functions

load_qualtrics_csv(→ list[dict[str, Any]])	Load the raw CSV file, and convert it to a flat dictionary. This function assumes row is the header, and skips rows 1 and 2 that are multi-index rows output from the Qualtrics platform.
validate_data(...)	Uses Pydantic model to convert and validate the raw Qualtrics data. Stream is validated together to ensure all recieved IDs exist within each of the component files.
create_dataframes(→ dict[str, pandas.DataFrame])	Creates the "base_questions" and "bridge_file" dataframes from the validated Pydantic model, output as a dictionary. Includes derivation of the deterministic global_id for the bridge_file.

Module Contents

bhf_smarthealth.parse.qualtrics_parser.logger

bhf_smarthealth.parse.qualtrics_parser.load_qualtrics_csv(file_path: pathlib.Path | str) → list[dict[str, Any]]

Load the raw CSV file, and convert it to a flat dictionary. This function assumes row is the header, and skips rows 1 and 2 that are multi-index rows output from the Qualtrics platform.

Parameters:

file_path (str | Path) – Path to the data file to be processed.

Returns:

A dictionary containing the raw, unvalidated Qualtrics data, with the extracted file metadata (sheffield_id and date) appended

Return type:

dict[str, Any]

bhf_smarthealth.parse.qualtrics_parser.validate_data(base_questions_data, bridge_data) → bhf_smarthealth.validate.qualtrics_schema.QualtricsStreamSchema

Uses Pydantic model to convert and validate the raw Qualtrics data. Stream is validated together to ensure all recieved IDs exist within each of the component files.

Parameters:

• base_questions_data (list[dict[str, Any]]) – List of dictionaries containing one row of the raw base questions data

• bridge_data (list[dict[str, Any]]) – List of dictionaries containing one row of the raw bridge file data

Returns:

Pydantic model class containing the validated data

Return type:

QualtricsStreamSchema

Raises:

DataValidationError – Raised if the pydantic schema validation fails

bhf_smarthealth.parse.qualtrics_parser.create_dataframes(validated_data: bhf_smarthealth.validate.qualtrics_schema.QualtricsStreamSchema) → dict[str, pandas.DataFrame]

Creates the “base_questions” and “bridge_file” dataframes from the validated Pydantic model, output as a dictionary. Includes derivation of the deterministic global_id for the bridge_file.

Parameters:

validated_data (QualtricsStreamSchema) – Pydantic model class containing the validated data

Returns:

Dictionary containing the “base_questions” and “bridge_file” dataframe

Return type:

dict[str, pd.DataFrame]