Imperfect representations of rural households: implications and improvements

Simon Fraval
Simon is a PhD candidate hosted by the International Livestock Research Institute and Wageningen University & Research. His research areas include ‘environmental impact assessment’ and ‘food and nutrition security’ in rural communities.

---

Each entry in the RHoMIS database provides a small insight into the life of a rural household. Research teams go to great lengths to make these snapshots in time as true to life as possible. We design the study – ‘localising’ the survey tool, setting an appropriate sample size and randomising the selection of households. We train a team of interviewers – providing a common understanding of each survey question. Then in implementing the survey, we travel long distances off the beaten track – by foot if necessary – having very personal discussions with rural households two to three times a day for weeks if not months at a time.

​Despite these efforts, some aspects of these ‘snapshots in time’ get distorted – resulting in imperfect representations of the rural households’ inputs, outputs, characteristics or wellbeing. 

If we are not aware of the imperfections of rural household data, then we run the risk of missing subtle changes over time and the risk of misunderstanding critical associations. It is these very insights that we need to use to inform policy and investment decisions for poverty alleviating rural development. 

In our recent peer-reviewed article we explore the occurrence and implications of ‘imperfect’ farm household survey data. Our article draws on three comparable multi-topic survey tools: LSMS-ISA, IMPACTlite and RHoMIS. Using these comparable survey tools, we assess the credibility of measurements, the reliability of measurements over time and the implications on setting an appropriate sample size for evidence-based decision making. Before getting into the details of our study, though, let’s look at how measurement error occurs in rural household surveys and what’s already been done to assess rural household data. 

The total measurement error of household survey data consists of random and systematic error. Random error can be thought of as instances where repeated measurements result in randomly inconsistent values, and systematic errors are consistently over or under-reported in a given measurement context. Measurement error can be introduced in each stage of the research process – design, training, implementation, the interview process and analysis.

​A random error, for example, could occur when a respondent tries their best to estimate something from a past event – say, the number of sacks filled after the most recent harvest. Similarly, a systematic error could arise in response to a sensitive question where there is a conscious or subconscious incentive to misreport; this could occur in matters such as infant feeding practices where the parent does not wish to appear negligent in the eyes of the interviewer.

​In the 1950s, the USDA pioneered the process of validating rural household data – comparing crop yields recalled by the farmer with measured crop yields.  More recently, the World Bank has instituted a validation program related to its LSMS-ISA implementations in low to middle-income countries.

​The LSMS-ISA, for example, has been used to validate the accuracy of land size estimates and test the effect of survey length on data quality. Several independent studies have also assessed measurement error, including gender bias and data fabrication (yes, that’s an interviewer making up the answers to an interview). ​

RHoMIS survey application, Senegal, Sep 2018

In our published article, we focus on the occurrence and influence of random error. First, we assess the credibility of maize yield and price data in comparison to historical yields and wholesale prices. We found that RHoMIS does not have the unrealistically high yield estimates seen in other datasets and unrealistic maize prices were similar between RHoMIS and LSMS-ISA. Both RHoMIS and LSMS-ISA had innovative cropping data collection strategies.

In the case of LSMS-ISA, enumerators visit households each cropping season – with the intention of minimising recall error. In the case of RHoMIS, households can quantify harvest volumes in a unit of their choice (such as standard sized sacks) rather than force kilogram estimates – minimising error due to respondent estimation. These innovations are positive; however, they are not sufficient to eliminate non-credible values.

In assessing the credibility of food security indicators, we found that RHoMIS produced more realistic results compared to LSMS-ISA and IMPACTlite – having lower instances of apparent starvation and lower instances of apparent excessive consumption. 

We also found some inconsistencies between survey rounds – including age of household head, household size and diet diversity. These results reinforce the notion that researchers need to consider their data collection strategy for each variable rather than assuming that some are ‘easy’ to enumerate.

In the instance of diet diversity, there were stark differences between IMPACTlite and RHOMIS – even though the same households were being interviewed. The differences between survey rounds may be explained by an unfortunate combination of question design and survey length. The data collection strategy of IMPACTlite was to ask an open question – “what food items did you consume?” – and allow the enumeration of a detailed list of food items. These questions on diet diversity, however, came at the end of more than two hours of questions and so the quality of data may have suffered from the farmer (and enumerator) being fatigued.

So, what are the implications of these errors in generating insights from rural household data? If the errors are random, then the sample sizes needed to detect substantial changes in the variables assessed in this study are in the hundreds – a readily attainable target. With tight controls on quality and a less variable population to represent, this sample size can be even smaller.

Rural sub-Saharan Africa is entering a stage of transformation where the opportunities and challenges for rural communities are becoming more pronounced. In this setting, the fundamentals of generating fit-for-purpose and representative observations remain a vital basis for informed decision making.

​For decision makers to make the most of such inherently coarse data, it is essential to have the foundation of robust sampling, quality-centric survey design (questions and length), transparency of experimental design and effective training. The quality and usability of such data can be further enhanced by improving coordination between agencies, incorporating mixed modes of data collection and continuing systematic validation programs.

Woman working on farm, Sierra Leone (Annie Spratt)