QA for merging

Purpose

This page summarizes which screening/demographics data and which home visit data have anomalies or missing data.

Screening/demographic data

screen_df <-
  readr::read_csv(
    file.path(
      here::here(),
      "data",
      "csv",
      "screening",
      "agg",
      "PLAY-screening-datab-latest.csv"
    ),
    col_types = readr::cols(.default = 'c'),
    show_col_types = FALSE
  )

The following rows have incomplete or missing site_id values:

screen_df |>
  dplyr::filter(is.na(site_id) | is.null(site_id)) |>
  dplyr::select(site_id, vol_id, participant_ID, session_id, play_id, group_name) |>
  dplyr::arrange(vol_id, site_id, participant_ID) |>
  knitr::kable() |>
  kableExtra::kable_classic()

site_id	vol_id	participant_ID	session_id	play_id	group_name
NA	1103	001	44638	NA	PLAY_Silver
NA	1481	006	64916	NA	PLAY_Silver
NA	1576	003	64939	NA	PLAY_Silver
NA	1656	001	70116	NA	PLAY_Silver
NA	954	001	39302	NA	PLAY_Silver

Volume 1103 is OHIOS. Volume 1482 is CSUFL. Volume 954 is GEORG. The missing values for vol_id indicate that there is a bug in the cleaning code.

2023-10-20

On closer investigation, the screening data do not show an OHIOS session with participant_ID == ‘001’. There are three with ‘000’ and two with ‘002’.

Similarly, for CSUFL, there is no ‘003’ or ‘006’. We have home visit data for ‘006’.

Similarly, for GEORG, there is a ‘???’, but no ‘001’.

The following rows have incomplete or missing participant_ID values:

screen_df |>
  dplyr::filter(is.na(participant_ID) | is.null(participant_ID)) |>
  dplyr::select(site_id, vol_id, participant_ID, session_id, play_id, group_name) |>
  dplyr::arrange(vol_id, site_id, participant_ID) |>
  knitr::kable() |>
  kableExtra::kable_classic()

site_id	vol_id	participant_ID	session_id	play_id	group_name
NA	NA	NA	NA	NA	NA
:——-	:——	:————–	:———-	:——-	:———-

The following rows have incomplete or missing play_id values:

screen_df |>
  dplyr::filter(is.na(play_id) | is.null(play_id)) |>
  dplyr::select(site_id, vol_id, participant_ID, session_id, play_id, group_name) |>
  dplyr::arrange(vol_id, site_id, participant_ID) |>
  knitr::kable() |>
  kableExtra::kable_classic()

site_id	vol_id	participant_ID	session_id	play_id	group_name
UCSCR	1066	001	56051	NA	PLAY_Gold
UCSCR	1066	002	56073	NA	PLAY_Gold
UCSCR	1066	003	56321	NA	PLAY_Gold
UCSCR	1066	005	57998	NA	PLAY_Gold
UCSCR	1066	009	58358	NA	PLAY_Gold
UCSCR	1066	010	58466	NA	PLAY_Gold
UCSCR	1066	011	58472	NA	PLAY_Gold
UCSCR	1066	012	58477	NA	PLAY_Gold
UCSCR	1066	014	58805	NA	PLAY_Silver
UCSCR	1066	015	59804	NA	PLAY_Gold
UCSCR	1066	016	59805	NA	PLAY_Gold
OHIOS	1103	002	56674	NA	PLAY_Gold
OHIOS	1103	002	56674	NA	PLAY_Gold
OHIOS	1103	005	57182	NA	PLAY_Gold
OHIOS	1103	006	58204	NA	PLAY_Gold
OHIOS	1103	008	58230	NA	PLAY_Gold
OHIOS	1103	009	57371	NA	PLAY_Gold
OHIOS	1103	010	57212	NA	PLAY_Gold
OHIOS	1103	011	57324	NA	PLAY_Gold
OHIOS	1103	012	58231	NA	PLAY_Gold
OHIOS	1103	014	58232	NA	PLAY_Gold
OHIOS	1103	015	58641	NA	PLAY_Gold
OHIOS	1103	016	58315	NA	PLAY_Gold
OHIOS	1103	017	58642	NA	PLAY_Gold
OHIOS	1103	018	58724	NA	PLAY_Gold
OHIOS	1103	019	58725	NA	PLAY_Gold
OHIOS	1103	021	58747	NA	PLAY_Gold
OHIOS	1103	023	59001	NA	PLAY_Gold
OHIOS	1103	024	59029	NA	PLAY_Gold
OHIOS	1103	025	59109	NA	PLAY_Gold
OHIOS	1103	026	59802	NA	PLAY_Gold
OHIOS	1103	027	59806	NA	PLAY_Gold
OHIOS	1103	028	59820	NA	PLAY_Gold
OHIOS	1103	029	59858	NA	PLAY_Gold
OHIOS	1103	030	59966	NA	PLAY_Gold
NA	1103	001	44638	NA	PLAY_Silver
STANF	1362	001	57209	NA	PLAY_Silver
STANF	1362	002	58017	NA	PLAY_Gold
PURDU	1363	001	56367	NA	PLAY_Silver
PURDU	1363	003	58740	NA	PLAY_Gold
PURDU	1363	004	58918	NA	PLAY_Gold
PURDU	1363	005	59150	NA	PLAY_Silver
PURDU	1363	006	60049	NA	PLAY_Silver
PURDU	1363	006	60049	NA	PLAY_Silver
CHOPH	1370	001	57863	NA	PLAY_Silver
CHOPH	1370	002	60897	NA	PLAY_Gold
CSULB	1376	001	56400	NA	PLAY_Gold
CSULB	1376	002	56399	NA	PLAY_Gold
CSULB	1376	003	57852	NA	PLAY_Gold
CSULB	1376	004	58612	NA	PLAY_Gold
CSULB	1376	005	59735	NA	PLAY_Silver
CSULB	1376	007	57857	NA	PLAY_Gold
CSULB	1376	010	60080	NA	PLAY_Gold
VBLTU	1391	001	59779	NA	PLAY_Silver
VBLTU	1391	003	60236	NA	PLAY_Gold
VBLTU	1391	004	60243	NA	PLAY_Gold
VBLTU	1391	005	60311	NA	PLAY_Gold
UHOUS	1397	001	57374	NA	PLAY_Silver
UHOUS	1397	002	57916	NA	PLAY_Gold
UHOUS	1397	004	58465	NA	PLAY_Gold
UHOUS	1397	005	59144	NA	PLAY_Gold
UHOUS	1397	006	60333	NA	PLAY_Gold
UHOUS	1397	007	61697	NA	PLAY_Gold
UHOUS	1397	008	61748	NA	PLAY_Gold
INDNA	1400	001	58458	NA	PLAY_Silver
INDNA	1400	002	62176	NA	PLAY_Gold
UIOWA	1422	001	57544	NA	PLAY_Gold
UIOWA	1422	002	58798	NA	PLAY_Gold
UIOWA	1422	003	59206	NA	PLAY_Gold
UIOWA	1422	004	59892	NA	PLAY_Gold
UIOWA	1422	005	60749	NA	PLAY_Silver
CSUFL	1481	001	60393	NA	PLAY_Silver
NA	1481	006	64916	NA	PLAY_Silver
NA	1576	003	64939	NA	PLAY_Silver
NA	1656	001	70116	NA	PLAY_Silver
NYUNI	899	001	41534	NA	PLAY_Gold
NYUNI	899	001	41534	NA	PLAY_Gold
NYUNI	899	002	41800	NA	PLAY_Silver
NYUNI	899	002	41800	NA	PLAY_Silver
NYUNI	899	003	41455	NA	PLAY_Silver
NYUNI	899	003	41455	NA	PLAY_Silver
NYUNI	899	003	41455	NA	PLAY_Silver
NYUNI	899	004	41535	NA	PLAY_Gold
NYUNI	899	005	41608	NA	PLAY_Gold
NYUNI	899	005	41608	NA	PLAY_Gold
NYUNI	899	006	41808	NA	PLAY_Gold
NYUNI	899	006	41808	NA	PLAY_Gold
NYUNI	899	007	41894	NA	PLAY_Gold
NYUNI	899	007	41894	NA	PLAY_Gold
NYUNI	899	013	43207	NA	PLAY_Gold
NYUNI	899	014	43530	NA	PLAY_Gold
NYUNI	899	017	55842	NA	PLAY_Gold
NYUNI	899	018	55863	NA	PLAY_Silver
NYUNI	899	020	56064	NA	PLAY_Silver
NYUNI	899	021	56065	NA	PLAY_Gold
NYUNI	899	022	56103	NA	PLAY_Gold
NYUNI	899	023	56104	NA	PLAY_Gold
NYUNI	899	024	56311	NA	PLAY_Silver
NYUNI	899	026	56417	NA	PLAY_Silver
NYUNI	899	028	56526	NA	PLAY_Gold
NYUNI	899	029	56571	NA	PLAY_Gold
NYUNI	899	031	57373	NA	PLAY_Silver
NYUNI	899	032	57410	NA	PLAY_Gold
NYUNI	899	035	57894	NA	PLAY_Gold
NYUNI	899	037	59428	NA	PLAY_Gold
NYUNI	899	227	38196	NA	PLAY_Gold
NYUNI	899	228	38197	NA	PLAY_Gold
NYUNI	899	229	38215	NA	PLAY_Gold
NYUNI	899	230	38236	NA	PLAY_Gold
NYUNI	899	231	38485	NA	PLAY_Gold
GEORG	954	002	40510	NA	PLAY_Silver
GEORG	954	002	40510	NA	PLAY_Silver
GEORG	954	003	41417	NA	PLAY_Gold
GEORG	954	003	41417	NA	PLAY_Gold
GEORG	954	004	41428	NA	PLAY_Gold
GEORG	954	005	41873	NA	PLAY_Gold
GEORG	954	005	41873	NA	PLAY_Gold
GEORG	954	005	41873	NA	PLAY_Gold
GEORG	954	008	42127	NA	PLAY_Gold
GEORG	954	008	42127	NA	PLAY_Gold
GEORG	954	009	42354	NA	PLAY_Gold
GEORG	954	009	42354	NA	PLAY_Gold
GEORG	954	010	42353	NA	PLAY_Gold
GEORG	954	011	42694	NA	PLAY_Gold
GEORG	954	012	57864	NA	PLAY_Gold
GEORG	954	015	58803	NA	PLAY_Gold
GEORG	954	016	59778	NA	PLAY_Silver
GEORG	954	021	60059	NA	PLAY_Gold
NA	954	001	39302	NA	PLAY_Silver
UCRIV	966	001	43624	NA	PLAY_Gold
VCOMU	982	002	42292	NA	PLAY_Silver
VCOMU	982	003	42940	NA	PLAY_Gold
UMIAM	996	003	65077	NA	PLAY_Gold

There are duplicate entries for OHIOS 1103 56674 002.

We should add duplicate checking to the cleaning code.

There are 1 variables with information about exclusion status:

ex_dups <- stringr::str_detect(names(screen_df), "exclusion")
names(screen_df)[ex_dups]

## [1] "exclusion_reason"

These probably result from some bug in the *_join operation in the cleaning process. They should be merged.

Home visit data

targets::tar_load(home_visit_df, store=file.path(here::here(), "_targets"))

The home_visit_df data have some field names that are inconsistent with the screening/demographic data files. We reconcile these differences first.

home_df <- home_visit_df |>
    dplyr::rename("play_id" = "participant_id") |>
    dplyr::rename("participant_ID" = "subject_number")

Since the home_visit_df data have not yet been merged with Databrary information, the vol_id and group_name variables are not available.

The following rows have incomplete or missing site_id values:

home_df |>
  dplyr::filter(is.na(site_id) | is.null(site_id)) |>
  dplyr::select(site_id, participant_ID, play_id) |>
  dplyr::arrange(site_id, participant_ID) |>
  knitr::kable() |>
  kableExtra::kable_classic()

site_id	participant_ID	play_id
NA	NA	NA
:——-	:————–	:——-

The following rows have incomplete or missing participant_ID values:

home_df |>
  dplyr::filter(is.na(participant_ID) | is.null(participant_ID)) |>
  dplyr::select(site_id, participant_ID, play_id) |>
  dplyr::arrange(site_id, participant_ID) |>
  knitr::kable() |>
  kableExtra::kable_classic()

site_id	participant_ID	play_id
NA	NA	NA
:——-	:————–	:——-

The following rows have incomplete or missing play_id values:

home_df |>
  dplyr::filter(is.na(play_id) | is.null(play_id)) |>
  dplyr::select(site_id, participant_ID, play_id) |>
  dplyr::arrange(site_id, participant_ID) |>
  knitr::kable() |>
  kableExtra::kable_classic()

site_id	participant_ID	play_id
NA	NA	NA
:——-	:————–	:——-

Home visit

MB-CDI