RHoMIS
  • Home
  • About
  • Survey
  • Data
  • Blog
  • Contact
  • Home
  • About
  • Survey
  • Data
  • Blog
  • Contact

Imperfect representations of rural households: implications and improvements

15/2/2019

0 Comments

 
Picture
Simon Fraval
Simon is a PhD candidate hosted by the International Livestock Research Institute and Wageningen University & Research. His research areas include ‘environmental impact assessment’ and ‘food and nutrition security’ in rural communities.


Each entry in the RHoMIS database provides a small insight into the life of a rural household. Research teams go to great lengths to make these snapshots in time as true to life as possible. We design the study – ‘localising’ the survey tool, setting an appropriate sample size and randomising the selection of households. We train a team of interviewers – providing a common understanding of each survey question. Then in implementing the survey, we travel long distances off the beaten track – by foot if necessary – having very personal discussions with rural households two to three times a day for weeks if not months at a time.

​Despite these efforts, some aspects of these ‘snapshots in time’ get distorted – resulting in imperfect representations of the rural households’ inputs, outputs, characteristics or wellbeing. 
If we are not aware of the imperfections of rural household data, then we run the risk of missing subtle changes over time and the risk of misunderstanding critical associations. It is these very insights that we need to use to inform policy and investment decisions for poverty alleviating rural development. 

In our recent peer-reviewed article we explore the occurrence and implications of ‘imperfect’ farm household survey data. Our article draws on three comparable multi-topic survey tools: LSMS-ISA, IMPACTlite and RHoMIS. Using these comparable survey tools, we assess the credibility of measurements, the reliability of measurements over time and the implications on setting an appropriate sample size for evidence-based decision making. Before getting into the details of our study, though, let’s look at how measurement error occurs in rural household surveys and what’s already been done to assess rural household data. 

The total measurement error of household survey data consists of random and systematic error. Random error can be thought of as instances where repeated measurements result in randomly inconsistent values, and systematic errors are consistently over or under-reported in a given measurement context. Measurement error can be introduced in each stage of the research process – design, training, implementation, the interview process and analysis.

​A random error, for example, could occur when a respondent tries their best to estimate something from a past event – say, the number of sacks filled after the most recent harvest. Similarly, a systematic error could arise in response to a sensitive question where there is a conscious or subconscious incentive to misreport; this could occur in matters such as infant feeding practices where the parent does not wish to appear negligent in the eyes of the interviewer.
​In the 1950s, the USDA pioneered the process of validating rural household data – comparing crop yields recalled by the farmer with measured crop yields.  More recently, the World Bank has instituted a validation program related to its LSMS-ISA implementations in low to middle-income countries.

​The LSMS-ISA, for example, has been used to validate the accuracy of land size estimates and test the effect of survey length on data quality. Several independent studies have also assessed measurement error, including gender bias and 
data fabrication (yes, that’s an interviewer making up the answers to an interview). ​
Picture
RHoMIS survey application, Senegal, Sep 2018
In our published article, we focus on the occurrence and influence of random error. First, we assess the credibility of maize yield and price data in comparison to historical yields and wholesale prices. We found that RHoMIS does not have the unrealistically high yield estimates seen in other datasets and unrealistic maize prices were similar between RHoMIS and LSMS-ISA. Both RHoMIS and LSMS-ISA had innovative cropping data collection strategies.

In the case of LSMS-ISA, enumerators visit households each cropping season – with the intention of minimising recall error. In the case of RHoMIS, households can quantify harvest volumes in a unit of their choice (such as standard sized sacks) rather than force kilogram estimates – minimising error due to respondent estimation. These innovations are positive; however, they are not sufficient to eliminate non-credible values.

In assessing the credibility of food security indicators, we found that RHoMIS produced more realistic results compared to LSMS-ISA and IMPACTlite – having lower instances of apparent starvation and lower instances of apparent excessive consumption. 

We also found some inconsistencies between survey rounds – including age of household head, household size and diet diversity. These results reinforce the notion that researchers need to consider their data collection strategy for each variable rather than assuming that some are ‘easy’ to enumerate.

In the instance of diet diversity, there were stark differences between IMPACTlite and RHOMIS – even though the same households were being interviewed. The differences between survey rounds may be explained by an unfortunate combination of question design and survey length. The data collection strategy of IMPACTlite was to ask an open question – “what food items did you consume?” – and allow the enumeration of a detailed list of food items. These questions on diet diversity, however, came at the end of more than two hours of questions and so the quality of data may have suffered from the farmer (and enumerator) being fatigued.

So, what are the implications of these errors in generating insights from rural household data? If the errors are random, then the sample sizes needed to detect substantial changes in the variables assessed in this study are in the hundreds – a readily attainable target. With tight controls on quality and a less variable population to represent, this sample size can be even smaller.
Rural sub-Saharan Africa is entering a stage of transformation where the opportunities and challenges for rural communities are becoming more pronounced. In this setting, the fundamentals of generating fit-for-purpose and representative observations remain a vital basis for informed decision making.

​For decision makers to make the most of such inherently coarse data, it is essential to have the foundation of robust sampling, quality-centric survey design (questions and length), transparency of experimental design and effective training. The quality and usability of such data can be further enhanced by improving coordination between agencies, incorporating mixed modes of data collection and continuing systematic validation programs.
Picture
Woman working on farm, Sierra Leone (Annie Spratt)
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    RHoMIS Blog

    The RHoMIS blog is written by a community of practice. The COP is made up of RHoMIS users and creators from across the world. Here we share their stories of how RHoMIS is helping to record and analyse household data.

    Archives

    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    December 2019
    November 2019
    October 2019
    September 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018

    Categories

    All
    Case Study
    Data
    Gender
    Innovation
    Milestone
    NGO
    Nutrition
    Research
    Rhomis
    Update
    Vietnam
    Workshop

    RSS Feed

Home
​​About
Team
​
​Use the Survey
Design
​DIY
Use the data
Examples
Science output

​Blog
Q&A
​

​Contact

RHoMIS - the Rural Household Multi-Indicator Survey
customisable questionnaire | digital infrastructure | advanced analysis

Used with over 30,000 households in 33 countries.




​​This Website is licensed under a Creative Commons Attribution 4.0 International License.