Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Survey Review Checklists #29

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,9 @@
# Keep tex and png in cloud resources training
!cloud-resources/**/*.tex
!cloud-resources/**/*.png


# Keep tex, png, and pdf in survey review checklists
!survey-review-checklists/**/*.tex
!survey-review-checklists/**/*.png
!survey-review-checklists/**/*.pdf
3 changes: 3 additions & 0 deletions survey-review-checklists/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# DIME Analytics - Survey Review Checklists

To update details
Binary file added survey-review-checklists/data_assurance.pdf
Binary file not shown.
184 changes: 184 additions & 0 deletions survey-review-checklists/data_assurance.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@

\documentclass{report}
\usepackage[most]{tcolorbox}

\usepackage{nopageno}
\usepackage{fancyhdr}
\usepackage{adjustbox}

\usepackage{booktabs,tabularx}
\usepackage{hyperref}
%\usepackage{showframe}
\usepackage{xcolor}
\usepackage{graphics}
\pagestyle{fancy}
\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\lfoot{\includegraphics[height=1cm,keepaspectratio]{img/i2i}}
\cfoot{\includegraphics[height=1cm,keepaspectratio]{img/wb}}
\rfoot{\includegraphics[height=1cm,keepaspectratio]{img/analytics}}
\definecolor{fontcolor}{HTML}{7A0569}
\newcommand{\titleBox}[1]{
\begin{tcolorbox}
[colframe = fontcolor,
colback = fontcolor,
sharp corners,
halign = flush center,
valign = center,
height = 0.2\textwidth,
after skip = 1cm]
#1
\end{tcolorbox}
}



%\input{checklist-header.tex}




% ------------------------------ End of preamble ---------------------------------------------
\begin{document}

\titleBox{
\textcolor{white}{\Large{DIME Analytics Data Quality Assurance Checklist}}
}

\section{Objective}
In the process of acquiring data, especially primary data, many factors could lead to poor data quality. A plan to ensure quality of data is one that identifies the potential issues in the data, and tracks data quality throughout the process of acquiring data.

\vspace{5mm} %5mm vertical space
\section*{Procurement}

\newcounter{row}

\newcommand{\makerow}[1]{%
% #1 = text field
#1 &
\stepcounter{row}%
\mbox{\CheckBox[print,name=YES\therow, width=0.7em, height=0.7em]{}} &
\mbox{\CheckBox[print,name=NO\therow, width=0.7em, height=0.7em]{}} &
\mbox{\CheckBox[print,name=N/A\therow, width=0.7em, height=0.7em]{}} \\
}


\begin{Form}
\noindent
\begin{tabularx}{\textwidth}{Xccc}
\toprule
\textbf{CHECKS} & YES & NO & N/A \\
\midrule
\makerow{Does the TOR establish \textbf{minimum quality indicators} required for the data to be considered acceptable?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure offhand what minimum quality indicators would be.

\midrule
\makerow{Does the TOR establish a \textbf{maximum response time} for the survey firm to address data quality issues identified?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps add, does the TOR establish a clear protocol for identifying and correcting data quality issues? maximum response time is part of that for sure, but establishing that the team will be running quality checks and the survey firm is expected to respond seems the most critical.

\bottomrule
\end{tabularx}

\vspace{5mm} %5mm vertical space

\section*{Survey Design}




\begin{tabularx}{\textwidth}{Xccc}
\toprule
\textbf{CHECKS} & YES & NO & N/A \\
\midrule
\makerow{Does the survey have \textbf{relevance and constraint logic} set up to aid the enumerator?}
\midrule
\makerow{Does the survey have an \textbf{an ID variable} to identify respondents and link them to the respondents database?}
\midrule
\makerow{Does the survey have an \textbf{unique ID generator} for each survey submission (if survey is being done over SurveyCTO this is automatically generated as the ‘key’ variable.}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems second-order to me (since ID var is already established)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is second-order, yes but comes in handy when dealing with duplicate submissions with the same ID variable as it allows for easy management of duplicates.

\midrule
\makerow{Does the survey have \textbf{duration spent} on completing a survey and modules within the survey?}
roshni13khincha marked this conversation as resolved.
Show resolved Hide resolved
\midrule

\makerow{Has the \textbf{translation} been approved?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by whom? maybe instead, was the survey fully translated into all languages used for data collection?

\midrule
\makerow{Does \textbf{enumerator training materials} include explanation of informed consent, Q\&A session, mock interviews and review of best practices?}
Copy link
Contributor

@mariaruth mariaruth Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest including something about the enumerator manual here as well, like does one exist and does it explain all questions and survey conventions. I see there is a follow up question about manuals but it reads as though it only applies for projects with multiple survey instruments.
maybe consolidate q&a/mock interviews to does the agenda for the enumerator training include both classroom and field practice, and allocate sufficient time for understanding the survey instrument?

\midrule
\makerow{In case of projects with multiple survey instruments, does each survey have it’s own detailed survey manual?}

\bottomrule
\end{tabularx}

\vspace{5mm} %5mm vertical space



\section*{High-Frequency Checks (HFC)}

\noindent
\begin{tabularx}{\textwidth}{Xcccc}
\toprule
\textbf{CHECKS} & YES & NO & N/A \\
\midrule
\makerow{Is there a plan in place to conduct \textbf{high-frequency checks} on the data being collected?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this question seems like the most important, the rest are good to document but not as critical. perhaps in general we want to think about setting this up as minimum expectation and then recommended best practices? not sure about the best way to do that, just thinking out loud ...

\midrule
\makerow{Does the HFC \textbf{monitor survey duration} and start/end times and day of the week of a survey fill-out?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does day of the week affect quality? maybe this is getting at more generally whether the HFCs monitor compliance with established field protocols re: interview duration and timing? might want to add here whether HFCs check for enumerator-specific patterns (e.g. an enumerator that is super slow or consistently doing interviews outside of expected working hours)

\midrule
\makerow{Do the HFCs check for key treatment and \textbf{outcome variables?}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check what for them? distribution?

\midrule
\makerow{Do the HFCs check for \textbf{sample completeness?}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meaning response rate / attrition?

\midrule
\makerow{Do the HFCs report \textbf{duplicates, missing values and errors} in logic that are not automatically corrected by the survey form?}
\midrule
\makerow{Is the process for generating flags from HFCs \textbf{easy and fast to run} on a daily basis for FCs?}
\midrule
\makerow{Does the HFC process account for potential lack of \textbf{access to WiFi} or electricity for team on the field?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is very specific. perhaps more generalizable, is there a protocol for sharing HFC results with teams that accounts for local conditions (e.g. limited connectivity) and accounts for safe sharing of personal data?

\midrule
\makerow{Do HFCs monitor geographic location of projects based on \textbf{GPS coordinates?}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this wording is quite vague. is it getting at whether HFCs check whether the interview was actually conducted in/near the expected location?

\midrule
\makerow{Has the HFC plan been communicated to all \textbf{stakeholders} in the team (incl. survey firm) with enough time before survey round?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could be incorporated with the question about protocol i suggested in my comment above.

\bottomrule
\end{tabularx}

\vspace{5mm} %5mm vertical space

\section*{Backchecks and audits}

\noindent
\begin{tabularx}{\textwidth}{Xcccc}
\toprule
\textbf{CHECKS} & YES & NO & N/A \\
\midrule
\makerow{Is there a plan in place to conduct \textbf{back-checks} on the data collected?}
Copy link
Contributor

@mariaruth mariaruth Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps have a space here for people to indicate what proportion of the sample was back-checked? might be simpler for that to be indicated explicitly rather than a yes/no in the question at the end. but maybe want to stick consistently with checklist.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do a combo of checklist plus spaces for open-ended responses in the reproducibility package checklist and that seems to work pretty well

\midrule
\makerow{Does the back-checks plan include \textbf{variables unlikely to change}, as well as variables that stem from difficult or lengthy modules? }
roshni13khincha marked this conversation as resolved.
Show resolved Hide resolved
\midrule
\makerow{Is there a plan to audit through \textbf{audio recordings}? If so, is it mentioned in the informed consent?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this question seems out of order because it comes in the middle of questions about backchecks

\midrule
\makerow{Does the back-checks plan include back-checking \textbf{all enumerators?}}
\midrule
\makerow{Does the back-check plan include back-checking \textbf{20 percent of the sample} in the first 2 weeks and 10 percent of the sample in the following weeks?}
\bottomrule
\end{tabularx}

\section*{Communication and Field Management}

\noindent
\begin{tabularx}{\textwidth}{Xcccc}
\toprule
\textbf{CHECKS} & YES & NO & N/A \\
\midrule
\makerow{Does team have a plan for \textbf{communication and data quality checks feedback} with enumerators?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems potentially redundant with the questions earlier about protocols for HFCs

\midrule
\makerow{Is there a protocol in place for the field team to \textbf{gather information and respond to flags} raised during data quality checks?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

potentially redundant with HFCs question, unless this is also meant to include backchecks/audits?

\midrule
\makerow{Has the team put together \textbf{tracking systems} for enumerators to fill out as part of their daily tasks?}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meaning like a supervisor logbook of interviews completed by the team?

\midrule
\makerow{Does the team have a plan for \textbf{reconciling data} from the tracking system with the responses on the server}
\midrule
\makerow{If feasible in the context, does the team plan on research staff \textbf{accompanying enumerator}? }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems like it would fit better in the sections on backchecks / audits

\bottomrule
\end{tabularx}


\end{Form}



\end{document}

Binary file added survey-review-checklists/img/analytics.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added survey-review-checklists/img/i2i.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added survey-review-checklists/img/wb.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.