bhf_smarthealth.log.metadata_logs#

This module contains functions to generate metadata logs.

Attributes#

logger

Classes#

DataQualityCounter

Custom collections.Counter instance to accept only allowed data quality categories as keys

Functions#

`evaluate_daily_processing_success`(counter[, ...])	Evaluate the data quality in a daily data batch. Logs information about the number of successfully-processed files if the specified threshold is satisfied, else an error is raised.
`generate_daily_IDs_receipt`(→ None)	Generate a .parquet receipt containing all of the sheffield_ids recieved on a single given date the specified data_type, and whether the data was valid on that date.

Module Contents#

bhf_smarthealth.log.metadata_logs.logger#

class bhf_smarthealth.log.metadata_logs.DataQualityCounter(iterable=None, /, **kwds)[source]#

Bases: collections.Counter[str]

Custom collections.Counter instance to accept only allowed data quality categories as keys

categories = ['missing', 'empty', 'valid', 'processing_error']#

bhf_smarthealth.log.metadata_logs.evaluate_daily_processing_success(counter: DataQualityCounter, validity_threshold: float = DEFAULT_VALID_DATA_THRESHOLD)[source]#

Evaluate the data quality in a daily data batch. Logs information about the number of successfully-processed files if the specified threshold is satisfied, else an error is raised.

Parameters:

counter (DataQualityCounter) – custom collections.Counter instance that accepts only allowed data quality category keys
validity_threshold (float, optional) – Ratio of invalid : total files allowed to pass through the pipeline, by default DEFAULT_VALID_DATA_THRESHOLD

Raises:

InsufficientValidDataError – Raised if an insufficient number of daily data files were successfully processed

bhf_smarthealth.log.metadata_logs.generate_daily_IDs_receipt(all_IDs: list[str], valid_IDs: list[str], data_stream: bhf_smarthealth.models.DataStreamNames, date_str: str, out_path: pathlib.Path) → None[source]#

Generate a .parquet receipt containing all of the sheffield_ids recieved on a single given date the specified data_type, and whether the data was valid on that date.