bhf_smarthealth.parse.gps_parser

bhf_smarthealth.parse.gps_parser#

This module contains functions to process raw GPS data ingressed to the Smart Health pipeline

Attributes#

logger

Functions#

load_gps_json(→ List[Dict[str, Any]])

Load GPS from JSON file provided by DAISER

validate_gps_data(...)

Validate the dictionaries containing the raw GPS data using Pydantic

gps_to_dataframe(→ pandas.DataFrame)

Convert raw GPS pydantic model to pandas DataFrame

add_gps_as_geometry(→ geopandas.GeoDataFrame)

Converts the raw GPS latitude and longitude to a geometry column

calculate_distances(→ geopandas.pd.Series)

Calculate distance between adjacent GeoDataFrame timepoints

calculate_speeds(→ geopandas.pd.Series)

Calculate speed of travel between adjacent GeoDataframe timepoints

filter_speeds(→ geopandas.GeoDataFrame)

Filter rows from a GeoDataframe where the speed of travel is unrealistic, i.e. less than zero or greater than the max_speed_ms

filter_time_jumps(→ geopandas.GeoDataFrame)

Nullify distance and speed for rows following a large time jump.

join_spatial_data(→ geopandas.GeoDataFrame)

Spatial join for GPS and reference data, by default a left join with a within predicate

nullify_lsoa_at_invalid_timepoints(...)

Set LSOA values to NULL for bad timepoints (i.e. distance is NULL)

Module Contents#

bhf_smarthealth.parse.gps_parser.logger#
bhf_smarthealth.parse.gps_parser.load_gps_json(file_path: str | pathlib.Path) → List[Dict[str, Any]][source]#

Load GPS from JSON file provided by DAISER

Parameters:

file_path (str | Path) – Path where JSON file is located

Returns:

List of dictionaries containing unvalidated, raw GPS datapoints with sheffield_id and date appended

Return type:

List[Dict[str, Any]]

Raises:

  • KeyError – Raised if the JSON file has an unexpected structure

  • FileNotFoundError – Raised if file not found

  • ValueError – Raised if file metadata cannot be validated

bhf_smarthealth.parse.gps_parser.validate_gps_data(gps_dict_list: List[Dict[str, Any]]) → bhf_smarthealth.validate.gps_schema.GPSDataList[source]#

Validate the dictionaries containing the raw GPS data using Pydantic

Parameters:

gps_dict_list (List[GPSDataPoint]) – List of dictionaries containing raw GPS datapoints

Returns:

Pydantic model class containing the validated GPS data

Return type:

GPSDataList

Raises:

ValidationError – Raised when input data is invalid

bhf_smarthealth.parse.gps_parser.gps_to_dataframe(time_series: bhf_smarthealth.validate.gps_schema.GPSDataList) → pandas.DataFrame[source]#

Convert raw GPS pydantic model to pandas DataFrame

Parameters:

time_series (GPSDataList) – Pydantic model class containing raw GPS data

Returns:

DataFrame containing raw GPS data

Return type:

pd.DataFrame

bhf_smarthealth.parse.gps_parser.add_gps_as_geometry(gps_df: pandas.DataFrame, crs: str = DEFAULT_GPS_CRS) → geopandas.GeoDataFrame[source]#

Converts the raw GPS latitude and longitude to a geometry column

Parameters:

  • gps_gdf (pd.DataFrame) – DataFrame containing raw GPS data

  • crs (str, optional) – Coordinate Reference System to use for GPS data, by default “EPSG:4326”

Returns:

DataFrame containing raw GPS data, with geometry column

Return type:

gpd.GeoDataFrame

Raises:

KeyError – Raised if DataFrame does not contain necessary latitude and/or longitude columns

bhf_smarthealth.parse.gps_parser.calculate_distances(gps_gdf: geopandas.GeoDataFrame, crs: str = DEFAULT_PLANAR_CRS, decimal_places: int = DEFAULT_DECIMAL_PLACES) → geopandas.pd.Series[source]#

Calculate distance between adjacent GeoDataFrame timepoints

Parameters:

  • gps_gdf (gpd.GeoDataFrame) – DataFrame containing raw GPS data, with geometry column

  • crs (str, optional) – Planar Coordinate Reference System to use for GPS data. Defaults to value set in config.py.

  • decimal_places (int, optional) – Number of decimal places to round calculated distance to. Defaults to value set in config.py. If invalid value in config.py, will be set to 3.

Returns:

Series containing calculated distances. Units are determined by the CRS (metres if using “EPSG:27700”, British National Grid)

Return type:

gpd.pd.Series

Raises:

  • KeyError – Raised if DataFrame does not contain necessary geometry column

  • ValueError – Raised if decimal_places in not a non-negative integer

Notes

  • The data is reprojected to a planar coordiante reference system for accurate distance calculations.

  • The first value will be NaN as there is no previous point for comparison

bhf_smarthealth.parse.gps_parser.calculate_speeds(gps_gdf: geopandas.GeoDataFrame, decimal_places: int = DEFAULT_DECIMAL_PLACES) → geopandas.pd.Series[source]#

Calculate speed of travel between adjacent GeoDataframe timepoints

Parameters:

  • gps_gdf (gpd.GeoDataFrame) – Dataframe containing raw GPS data, with a datetime index and a distance column

  • decimal_places (int, optional) – Number of decimal places to round calculated distance to. Defaults to value set in config.py. If invalid value in config.py, will be set to 3.

Returns:

Series containing calculated speeds. Units are XX per second, where XX is the unit of distance in the distance column

Return type:

gpd.pd.Series

Raises:

  • KeyError – Raised if distance column is missing.

  • AttributeError – Raised if DataFrame does not have a datetime index

  • ValueError – Raised if decimal_places in not a non-negative integer

Notes

  • The first value will be NaN as there is no previous point for comparison

bhf_smarthealth.parse.gps_parser.filter_speeds(gps_gdf: geopandas.GeoDataFrame, max_speed_ms: float = DEFAULT_MAX_SPEED_MS) → geopandas.GeoDataFrame[source]#

Filter rows from a GeoDataframe where the speed of travel is unrealistic, i.e. less than zero or greater than the max_speed_ms

Parameters:

  • gps_gdf (gpd.GeoDataFrame) – Dataframe containing raw GPS data, with a speed column

  • max_speed_ms (float, optional) – Maximum accepted speed in metres per second. Defaults to value set in config.py.

Returns:

Cleaned Dataframe

Return type:

gpd.GeoDataFrame

Raises:

KeyError – Raised if speed column not in GeoDataframe

Notes

  • NaN values are not filtered out

bhf_smarthealth.parse.gps_parser.filter_time_jumps(gps_gdf: geopandas.GeoDataFrame, max_jump_mins: int = DEFAULT_MAX_TIME_JUMP_MINS) → geopandas.GeoDataFrame[source]#

Nullify distance and speed for rows following a large time jump.

Parameters:

  • gps_gdf (gpd.GeoDataFrame) – Dataframe containing raw GPS data, with datetime index

  • max_jump_mins (int, optional) – Maximum acceptable time jump in whole minutes. Defaults to value set in config.py.

Returns:

Cleaned Dataframe

Return type:

gpd.GeoDataFrame

Raises:

AttributeError – Raised if index is not a datetime index

Notes

This function sets distance and speed to NaN rather than removing rows as the location (LSOA) at that timepoint is still valid data

bhf_smarthealth.parse.gps_parser.join_spatial_data(gps_gdf: geopandas.GeoDataFrame | pandas.DataFrame, ref_data: Any, nan_replace_values: dict[str, str] | None = None, **kwargs: Any) → geopandas.GeoDataFrame[source]#

Spatial join for GPS and reference data, by default a left join with a within predicate

Parameters:

  • gps_gdf (gpd.GeoDataFrame) – GPS GeoDataFrame

  • ref_data (Any) – Reference data to spatial join to GPS data (most likely LSOA data)

Returns:

GPS GeoDataFrame with reference data joined

Return type:

gpd.GeoDataFrame

Raises:

KeyError – Raised if the DataFrame doesn’t have a geometry column

bhf_smarthealth.parse.gps_parser.nullify_lsoa_at_invalid_timepoints(df: geopandas.GeoDataFrame | pandas.DataFrame, lsoa_cols: list[str] = ['LSOA21CD', 'LSOA21NM'], drop_null_rows: bool = True) → geopandas.GeoDataFrame | pandas.DataFrame[source]#

Set LSOA values to NULL for bad timepoints (i.e. distance is NULL)

Parameters:

  • df (gpd.GeoDataFrame | pd.DataFrame) – GPS GeoDataFrame or DataFrame

  • lsoa_cols (List[str], optional) – List of column names containing LSOA data

  • drop_null_rows (bool , optional) – Optionally drop the rows where the distance and LSOA values are NULL

Returns:

GPS GeoDataFrame or DataFrame

Return type:

gpd.GeoDataFrame | pd.DataFrame

Raises:

KeyError – Raised if distance or LSOA columns are not present

On this page