• No one best way: decision depends on:

–        What you need to know: numbers or stories

–        Where the data reside: environment, files, people

–        Resources and time available

–        Complexity of the data to be collected

–        Frequency of data collection

–        Intended forms of data analysis

Quantitative Approach

• Data in numerical form
• Data that can be precisely measured

–        age, cost, length, height, area, volume, weight, speed, time, and temperature

• Harder to develop
• Easier to analyze

Qualitative Approach

• Data that deal with description
• Data that can be observed or self-reported, but not always precisely measured
• Less structured, easier to develop
• Can provide “rich data” — detailed and widely applicable
• Is challenging to analyze
• Is labor intensive to collect
• Usually generates longer reports

Data Collection Tools

1. Participatory Methods means Involve groups or communities heavily in data collection Examples:

o   community meetings means One of the most common participatory methods Must be well organized , agree on purpose, establish ground rules, who will speak,time allotted for speakers ,format for questions and answers

o   mapping  means Drawing or using existing maps, Useful tool to involve stakeholders, increases understanding of the community, generates discussions, verifies secondary sources of information, perceived changes, Types of mapping:, natural resources, social, health, individual or civic assets, wealth, land use, demographics

o   transect walks means Evaluator walks around community observing people, surroundings, and resources, Need good observation skills, Walk a transect line through a map of a community — line should go through all zones of the community

1. Observation

Ways to Record Information from Observations

• Observation guide

–        printed form with space to record

• Recording sheet or checklist

–        Yes/no options; tallies, rating scales

• Field notes

–        least structured, recorded in narrative, descriptive style

Observation is Helpful when:

• need direct information
• trying to understand ongoing behavior
• there is physical evidence, products, or outputs than can be observed
• need to provide alternative when other data collection is infeasible or inappropriate
• Structured: determine, before the observation, precisely what will be observed before the observation
• Unstructured: select the method depending upon the situation with no pre-conceived ideas or a plan on what to observe
• Semi-structured: a general idea of what to observe but no specific plan

Guidelines for Planning Observations

• Have more than one observer, if feasible
• Train observers so they observe the same things
• Pilot test the observation data collection instrument
• For less structured approach, have a few key questions in mind

•          Advantages	•          Collects data on actual vs. self- reported behavior or perceptions. It is real-time vs. retrospective
•          Challenges	•          Observer bias, potentially unreliable; interpretation and coding challenges; sampling can be a problem; can be labor intensive; low response rates

1. Surveys and Interviews

• Excellent for asking people about:

o   perceptions, opinions, ideas

• Less accurate for measuring behavior
• Sample should be representative of the whole
• Big problem with response rates

Structures for Surveys

1. Structured:
   1. Precisely worded with a range of pre-determined responses that the respondent can select
   2. Everyone asked exactly the same questions in exactly the same way, given exactly the same choices
2. Semi-structured
   1. Asks same general set of questions but answers to the questions are predominantly open-ended

Structured	harder to develop easier to complete easier to analyze more efficient when working with large numbers
Semi-structured	easier to develop: open ended questions more difficult to complete: burdensome for people to complete as a self-administrated questionnaire harder to analyze but provide a richer source of data, interpretation of open-ended responses subject to bias

Modes of Survey Administration

1. a)Telephone surveys
2. b)Self-administered questionnaires distributed by mail, e-mail, or websites
3. c)Administered questionnaires, common in the development context
4. d)In development context, often issues of language and translation

Advantages and Challenges of Surveys

Advantages	Best when you want to know what people think, believe, or perceive, only they can tell you that
Challenges	People may not accurately recall their behavior or may be reluctant to reveal their behavior if it is illegal or stigmatized. What people think they do  or say they do is not always the same as what they actually do.

Interviews

• Often semi-structured
• Used to explore complex issues in depth
• Forgiving of mistakes: unclear questions can be clarified during the interview and changed for subsequent interviews
• Can provide evaluators with an intuitive sense of the situation

Challenges of Interviews

• Can be expensive, labor intensive, and time consuming
• Selective hearing on the part of the interviewer may miss information that does not conform to pre-existing beliefs
• Cultural sensitivity: e.g., gender issues

1. Focus Groups

• Type of qualitative research where small homogenous groups of people are brought together to informally discuss specific topics under the guidance of a moderator
• Purpose: to identify issues and themes, not just interesting information, and not “counts”

Focus Groups Are Inappropriate when:

• language barriers are insurmountable
• evaluator has little control over the situation
• trust cannot be established
• free expression cannot be ensured
• confidentiality cannot be assured

Focus Group Process

Phase		Action
1	Opening	Ice-breaker; explain purpose; ground rules; introductions
2	Warm-up	Relate experience; stimulate group interaction; start with least threatening and simplest questions
3	Main body	Move to more threatening or sensitive and complex questions; elicit deep responses; connect emergent data to complex, broad participation
4	Closure	End with closure-type questions; summarize and refine; present theories, etc; invite final comments or insights; thank participants

Advantages and Challenges of Focus Groups

Advantages	Can be conducted relatively quickly and easily; may take less staff time than in-depth, in-person interviews; allow flexibility to make changes in process and questions; can explore different perspectives; can be fun
Challenges	Analysis is time consuming; participants not be representative of population, possibly biasing the data; group may be influenced by moderator or dominant group members

1. Diaries, Journals, Self-reported Checklists

• Use when you want to capture information about events in people’s daily lives
• Participants capture experiences in real-time not later in a questionnaire
• Used to supplement other data collection

Guidelines for Diaries or Journals

Step	Process
1	Recruit people face-to-face • encourage participation, appeal to altruism, assure confidentiality, provide incentive
2	Provide a booklet to each participant • cover page with clear instructions, definitions, example • short memory-joggers, explain terms, comments on last page , calendar
3	Consider the time-period for collecting data • if too long, may become burdensome or tedious • if too short may miss the behavior or event

Advantages and Challenges of Diaries and Self-reported Checklists

Advantages	Can capture in-depth, detailed data that might be otherwise forgotten Can collect data on how people use their time Can collect sensitive information Supplements interviews provide richer data
Challenges	Requires some literacy May change behavior Require commitment and self-discipline Data may be incomplete or inaccurate Poor handwriting, difficult to understand phrases

1. Expert Judgment

Use of experts, one-on-one or as a panel E.g., Government task forces, Advisory Groups

Selecting Experts

• Establish criteria for selecting experts not only on recognition as expert but also based on:

–        areas of expertise

–        diverse perspectives

–        diverse political views

–        diverse technical expertise

Advantages	Fast, relatively inexpensive
Challenges	Weak for impact evaluation May be based mostly on perceptions Value of data depends on how credible the experts are perceived to be

1. Delphi Technique means Enables experts to engage remotely in a dialogue and reach consensus, often about priorities

• Experts asked specific questions; often rank choices
• Responses go to a central source, are  summarized and fed back to the experts without attribution
• Experts can agree or argue with others’ comments
• Process may be iterative

Advantages	Allows participants to remain anonymous Is inexpensive Is free of social pressure, personality influence, and individual dominance Is conducive to independent thinking Allows sharing of information
Challenges	May not be representative Has tendency to eliminate extreme positions Requires skill in written communication Requires time and participant commitment