Needs Analysis

1. What is Needs Analysis?
2. Stages of Needs analysis
   1. Stakeholder Identification
   2. Data collection
   3. Analysis
3. Data collection for Needs analysis
   1. Sampling strategies for quantitative surveys
   2. Sampling strategies for qualitative surveys

1. What is Needs Analysis?

Needs analysis is used to clarify a project’s aims and to ensure that it is appropriate for the context within which it will be delivered. A successful needs analysis increases the likelihood of project success by ensuring that a project:

• Responds to the real needs of its target beneficiaries (i.e. target individuals, communities or organisations)
• Addresses a real service gap
• Is informed by an understanding of the specific context in which it is delivered
• Is sensitive to the values, aims and priorities of those it is intended to benefit
• Engages and elicits ‘buy in’ from all relevant stakeholders
• Takes appropriate account of context-specific resources and barriers
• Is appropriately designed, feasible and likely to be cost-effective with a well-developed theory of change that draws from data on the scale, consequences, nature and causes of the problem or gap it seeks to address

2. Stages of Needs Analysis

1. 1. Stakeholder identification: Determine all groups, organisations or individuals who have a “stake” in the project’s delivery and/or outcomes.
      • Who is intended to benefit from the project?
      • Does this population include subgroups?
      • Who will deliver the project?
      • Are there any other organisations delivering similar projects?
      • Are there any organisations with whom the project will need to liaise or cooperate with in order to succeed?
      • Are there any government agencies with a stake in the project’s delivery and/or outcomes (e.g. who could benefit from, support or derail the project)?
      • Does the project’s success depend on the activities or services provided by others?
      • Can contacted stakeholders identify any other stakeholders that may have been overlooked?
   2. Data Collection: Use focus groups, public administration data and/or survey methods to accurately map stakeholders’ views on:
      • what their core needs are
      • how they prioritise their core needs
      • which needs are not being met
      • which needs they believe the project could (or should) address and how
      • what they could gain and/or contribute to the project
      • how they would like their interests to be represented during the project planning and implementation.
   3. Analysis: Data obtained through stakeholder consultation can be analysed in a variety of ways. Quantitative data from surveys and/or administrative datasets requires statistical analysis, with attention to sampling strategies needed to ensure that results are generaliseable to the relevant population. Qualitative data from focus groups and consultations are usually analysed thematically to map different stakeholders’ experiences and perspectives. Qualitative and quantitative data are then drawn together to:
      • map needs and service gaps
      • prioritise needs and gaps
      • develop a project theory of change
      • design a feasible and acceptable project design

Gap analysis is usually used to address these dimensions using a variety of data. The gap analysis brings together the following information:

1. • Data about target populations identified from secondary data (which may include administrative data) and grassroots contacts; in large-scale projects, data would also be collected using surveys and qualitative interviews or focus groups. This usually includes demographic information, abilities, financial means etc.
   • Findings about met and unmet needs from the needs analysis, and used to refine or adapt the project’s problem definition
   • Information about existing services and resources, funding, and populations served (e.g. teacher-student ratios, available materials, barriers or shortfalls in service provision)
   • Secondary data about availability, accessibility, and appropriateness of existing services for the target population. This usually includes data on outcomes of interest under the current regime (e.g. science achievement measures; rates of science enrolment or literacy; number of students going on to post-graduate science studies; etc.)

3. Data Collection for Needs Analysis

OAD projects will not typically engage in extensive data collection on their own. Collecting social data is full of unexpected challenges; it is easy to inadvertently collect misleading data or to spend a lot of time, energy and expense collecting data which later proves useless because of some small oversight. If you are interested in collecting data, however, we hope to support you in doing so—ideally in partnership with professionals who specialise in this area (e.g. economists, statisticians, and/or development organisations). The following sections outline what is involved so that you can determine whether it is something that might interest you or not.

Important points
Points to Note About Survey Data

Data collection requires careful planning regarding sampling strategies and the development of data collection tools. For quantitative data (e.g. surveys), questionnaires must be carefully designed (see accompany guide with template questions). There is extensive research showing how small changes in wording or presentation can significantly alter the results you get. Advice and expert peer-review are therefore important safeguards in the actual survey design stage.

It is ideal to have someone who has been trained and is familiar with the questionnaire administer surveys in person or over the telephone. In many places, there are professionals or research firms who will provide this service for a reasonable fee. If a survey is important to you, consider including this in your project budget.

Trained survey administrators ensure that respondents have answered all of the questions in a legible and consistent fashion. They thus minimise the amount of missing or incorrect data. For example, if a respondent would prefer not to answer a question, a survey administrator will ensure that ‘Prefer not to say’ is ticked (or noted) on the survey instrument rather than leaving that item blank (or, worse, sometimes selecting ‘Prefer not to say’ and sometimes leaving it blank). If the item is left blank, it is unclear whether the respondent withheld the information (which is informative), simply didn’t notice the question or whether the administrator themselves missed the question.

Correctly recording the reasons for missing data and minimising the amount of missing data are important because missing data quickly invalidate survey results. When individuals are asked to complete and return a survey themselves, levels of missing data are much higher than they would otherwise be and can make the entire data collection process useless (including both surveys that are missing in their entirety and surveys that are returned but include missing questions/items).

Points to Note About Interviews and Focus Groups

For qualitative data (i.e. focus groups and/or interviews), data collectors need topic guides that outline which topics to cover during a session. Ideally, qualitative data are recorded with a dictaphone and later transcribed. This is, however, time and resource intensive: one hour of recorded data takes approximately 3 hours to transcribe into text. Texts then need to be analysed to determine which themes recur. When resources are limited, it is more efficient not to record the session but to have it conducted by two assessors: one to guide the discussion while the others takes notes. Notes from each session can then be compiled to inform the needs assessment.

Sampling Strategies

Different sampling strategies are used for quantitative (survey, questionnaire) and qualitative (interview, focus group) data collection.

Quantitative (Survey) Sampling

Quantitative data are used to describe populations quantitatively: that is, to be able to accurately quantify the percentage of individuals within a population who share a specific characteristic. Surveys can offer important information on attitudes, beliefs, perceptions, level of satisfaction, needs, priorities, service access and so on which can be used to inform project theory, design, resourcing, and implementation planning. For example, if 95% of a community say that they use mobile phone data on a daily basis, a project using a mobile-based information dissemination mechanisms is more likely to be successful than it would be in a community where only 45% of individuals have access to mobile data.

Surveys are relatively easy to analyze and provide anonymity to respondents. However, in order to generate usable quantitative information, surveys must:

• Use closed questions: Individuals need to be asked the same questions and in the same way. In order to be meaningfully analysed using statistical methods, their responses also need to fit into discrete categories. Survey questionnaires are therefore comprised of closed questions (i.e. asking participants to select a category, offer a number, rank items or select a value on a specified scale).
• Use a representative sampling strategy: There are two ways to achieve statistically representative samples from a population of interest: (1) obtain data from everyone in the population or (2) obtain data from a random sample selection (or the statistical equivalent of a random selection).

It is usually too expensive and/or impractical to gather data from every individual in a community or other target population. Surveys therefore usually rely on data from a smaller group of individuals drawn from within the larger population of interest. It is important that this smaller sample is representative of the larger population.  If not, survey data will be misleading: the percentages of individuals reporting a certain need or perspective may then fail to reflect the true percentage in the population and lead to a range of project miscalculations and resourcing errors. In order to be representative, a survey sample needs to be drawn at random (or the statistical equivalent) from the larger population.

Both of these survey features can be challenging to implement in practice. This leads to short cuts that actually produce invalid (and probably misleading) results. There are many examples of surveys that do not fulfil these basic criteria for validity in the social science literature on the Astronomy community (e.g. of gender composition and gender differences; and of Astronomy capacity and research output). In fact, we request that anyone who finds a properly conducted random sample  survey of astronomers please let us know!

Survey questionnaire design can be simplified by using and adapting a template, with minimal changes made to template question formats. Random sampling is more difficult in practice. In the ideal case, administrative data are used to compile a full list of population members, with survey participants selected at random from this list and then contacted. Expert advice is likely to be need to construct alternative valid sampling strategies.

Qualitative Sampling

Sampling is more straight-forward for qualitative data collection. Purposive sampling is typically used, in which those conducting the needs assessment deliberately seek out key individuals or groups based on characteristics of interest. A purposive community sample would include community leaders of both genders and from different community segments as well as a small number of community members drawn from each identifiable subgroup within the community. Effort is typically made to seek out different perspectives on multiple dimensions (e.g. interviewing community members with varying economic status and of both genders from within each ethnic group).

Making Use of Secondary Datasets

Secondary datasets comprise, for example, World Bank or OECD development data. Some links to get started exploring these sources can be found here.