Simon is a PhD candidate hosted by the International Livestock Research Institute and Wageningen University & Research. His research areas include ‘environmental impact assessment’ and ‘food and nutrition security’ in rural communities.
Each entry in the RHoMIS database provides a small insight into the life of a rural household. Research teams go to great lengths to make these snapshots in time as true to life as possible. We design the study – ‘localising’ the survey tool, setting an appropriate sample size and randomising the selection of households. We train a team of interviewers – providing a common understanding of each survey question. Then in implementing the survey, we travel long distances off the beaten track – by foot if necessary – having very personal discussions with rural households two to three times a day for weeks if not months at a time.
Despite these efforts, some aspects of these ‘snapshots in time’ get distorted – resulting in imperfect representations of the rural households’ inputs, outputs, characteristics or wellbeing.
If we are not aware of the imperfections of rural household data, then we run the risk of missing subtle changes over time and the risk of misunderstanding critical associations. It is these very insights that we need to use to inform policy and investment decisions for poverty alleviating rural development.
In our recent peer-reviewed article we explore the occurrence and implications of ‘imperfect’ farm household survey data. Our article draws on three comparable multi-topic survey tools: LSMS-ISA, IMPACTlite and RHoMIS. Using these comparable survey tools, we assess the credibility of measurements, the reliability of measurements over time and the implications on setting an appropriate sample size for evidence-based decision making. Before getting into the details of our study, though, let’s look at how measurement error occurs in rural household surveys and what’s already been done to assess rural household data.
The total measurement error of household survey data consists of random and systematic error. Random error can be thought of as instances where repeated measurements result in randomly inconsistent values, and systematic errors are consistently over or under-reported in a given measurement context. Measurement error can be introduced in each stage of the research process – design, training, implementation, the interview process and analysis.
A random error, for example, could occur when a respondent tries their best to estimate something from a past event – say, the number of sacks filled after the most recent harvest. Similarly, a systematic error could arise in response to a sensitive question where there is a conscious or subconscious incentive to misreport; this could occur in matters such as infant feeding practices where the parent does not wish to appear negligent in the eyes of the interviewer.
In our published article, we focus on the occurrence and influence of random error. First, we assess the credibility of maize yield and price data in comparison to historical yields and wholesale prices. We found that RHoMIS does not have the unrealistically high yield estimates seen in other datasets and unrealistic maize prices were similar between RHoMIS and LSMS-ISA. Both RHoMIS and LSMS-ISA had innovative cropping data collection strategies.
In the case of LSMS-ISA, enumerators visit households each cropping season – with the intention of minimising recall error. In the case of RHoMIS, households can quantify harvest volumes in a unit of their choice (such as standard sized sacks) rather than force kilogram estimates – minimising error due to respondent estimation. These innovations are positive; however, they are not sufficient to eliminate non-credible values.
In assessing the credibility of food security indicators, we found that RHoMIS produced more realistic results compared to LSMS-ISA and IMPACTlite – having lower instances of apparent starvation and lower instances of apparent excessive consumption.
We also found some inconsistencies between survey rounds – including age of household head, household size and diet diversity. These results reinforce the notion that researchers need to consider their data collection strategy for each variable rather than assuming that some are ‘easy’ to enumerate.
In the instance of diet diversity, there were stark differences between IMPACTlite and RHOMIS – even though the same households were being interviewed. The differences between survey rounds may be explained by an unfortunate combination of question design and survey length. The data collection strategy of IMPACTlite was to ask an open question – “what food items did you consume?” – and allow the enumeration of a detailed list of food items. These questions on diet diversity, however, came at the end of more than two hours of questions and so the quality of data may have suffered from the farmer (and enumerator) being fatigued.
So, what are the implications of these errors in generating insights from rural household data? If the errors are random, then the sample sizes needed to detect substantial changes in the variables assessed in this study are in the hundreds – a readily attainable target. With tight controls on quality and a less variable population to represent, this sample size can be even smaller.
The RHoMIS community of practice is made up of RHoMIS users and creators from across the world. Here we share their stories of how RHoMIS is helping to record and analyse household data.