I. PURPOSE OF THE EVALUATION

Development Solutions (DS), an implementing partner for the USAID/Mongolia’s Business Excellence for Sustainability and Transparency (BEST) program, requires a comprehensive, evidence-based, and independent final performance evaluation of its program. The evaluation report must provide both qualitative and quantitative assessments of the progress made towards achieving the expected program results. This performance evaluation aims to assess the BEST program’s immediate outcomes and overall performance in Ulaanbaatar and 12 provinces of Mongolia, covering the period from December 2, 2019, to June 30, 2024. In particular, the evaluation will cover two (2) main areas:

• The systematic review of the BEST program’s development hypothesis and its implementation approach linked together all five components of the Program Cycle, paying attention to its results measured against its expected objectives.
• Key lessons learned, conclusions, and related recommendations to consider for similar activities

The purpose of this final evaluation is twofold: to acquire insights into enhancing development outcomes and to foster accountability and learning from implementing the BEST program. This final performance evaluation should systematically analyze USAID’s BEST program outcomes to inform decisions about current and future similar programs for further improvement and provide necessary recommendations at the implementation and decision-making levels.

The purpose of the performance evaluation is to:

• Examine to what extent the assumed causal relationships between activities, outputs, outcomes, and impacts in the theory of change were supported by evidence.
• Assess the achievement of the BEST program’s intended outcomes in Mongolia’s private sector/SMEs in the selected regions.
• Analyze how the BEST program navigated and adapted to unforeseen challenges and the evolving needs of SMEs.
• Evaluate how effectively the program activities and approaches addressed gender differences and gaps.
• Assess the sustainability of the BEST program outcomes after the program ends and evaluate the approaches or strategies implemented to ensure this sustainability.

The Evaluator must not simply provide an accounting of performance against targets but offer an independent analysis of why targets were realized, met, or significantly surpassed. Additionally, the Evaluator should also examine how effectively the program adapted to unforeseen challenges, particularly during the COVID-19 pandemic, and how well it addressed gender differences and gaps, including the integration of women-owned SMEs’ needs. Furthermore, the evaluation should consider the unintended outcomes and their contributions to the program’s effectiveness and provide an assessment of the sustainability of program outcomes and the strategies implemented to ensure long-term impacts.

The results of this evaluation will inform DS of implementing capacity-building activities central to the BEST program’s strategic approach to strengthening private sector development. The audience of the evaluation may include, but is not limited to, USAID/Mongolia and key stakeholders such as the Ministry of Agriculture and Light Industry (MoFALI), the Small and Medium Enterprises (SME) Agency of Mongolia, other national and local government institutions, development agencies, and banking and financial institutions.

Evaluation findings will also contribute to the substantial evidence on good practices in SME development, highlighting areas that can be amplified, replicated, and/or sustained and the enabling or hindering factors affecting the performance and results of similar private-sector-focused programs and activities.

II. PROGRAM INFORMATION

III. BACKGROUND

USAID/Mongolia’s grant to Development Solutions for the five-year BEST program was intended to help Mongolia’s economy grow and diversify by increasing lending to SMEs and improving economic governance and accountability. The development hypothesis of this activity is that increased access to finance by SMEs and improved corporate governance will spread equality of opportunity and tap the full potential of the economy.

The BEST program objectives will be achieved through five Objectives:

• Objective 1. Increase SME access to finance
• Objective 2. Improve corporate governance among SMEs
• Objective 3. Improve capacity within Mongolian institutions
• Objective 4. Maintain an analytical agenda for strategic learning, and
• Objective 5. Provide small grants to reach disadvantaged but otherwise well-qualified small enterprises.

The intended outcomes of the program include the following:

1. Increased access to finance for SMEs

                  a) SMEs strengthened to qualify for loans

                  b) Loans facilitated through program partnerships

     2. Improved corporate governance among SMEs

                  a) SMEs adopted good governance practices

    3. Well-established and transparent public resource mobilization

                  a) Strengthened province-level NGOs advocating for SMEs interests

                  b) Improved capacity for safeguarding against waste, fraud, and corruption among government officials

    4. Evidence-based decision-making and SME policy development

                 a) More accurate understanding of the needs and performance of the private sector

   5. Accelerated growth of disadvantaged enterprises

               a) More financially secure and strengthened disadvantaged enterprises

RESULTS FRAMEWORK: An overarching Results Framework was developed to explain how the BEST program’s strategic approaches will address the constraints SMEs face in access to finance and long-term sustainability. The framework demonstrates causality and the logical linkages between shorter-term, intermediate, and longer-term results. The Results Framework is founded upon the following overarching theory of change:

IF the BEST program improves access to finance and corporate governance among SMEs through the loan facilitation and capacity building interventions THEN they will contribute more fully to the generation of jobs and increase in economic wealth.

IF local level NGOs effectively serve as a watchdog role that advocates for a transparent and fair financing environment THEN it will lead to better and efficient mobilization of government funded financing.

IF the BEST program promotes availability of statistical evidence on the state of SMEs, THEN it will help to achieve more accurate diagnoses of the needs and performance of SMEs, thus design better SME development policies.

In the period covered by this evaluation, the BEST program reached more than 15,000 beneficiaries^[1] (SMEs, local NGOs, and government officials) in 13 program locations through training, loan facilitation services, consultancy and coaching services, and a small grant sub-program.

The BEST program’s primary activities that work toward intended outcomes are:

1.  Loan facilitation service for SMEs: The BEST program provides technical assistance on loan materials to ensure SME loan applications comply with the financial institutions’ requirements. Through the assistance provided, SMEs submit their loan applications to the appropriate financial institution that will potentially facilitate the desired loan. During the program lifecycle of five-year implementation, 7,730 SME loan applications were prepared and refined with BEST assistance, of which 4,949 were approved for loans worth a total of MNT 284,674,958,079 (US$84,192,722).
2. Capacity-building training for SMEs: BEST provides capacity-building training on financial and accounting literacy, loan literacy, business plan development, and good governance for the beneficiary SMEs working in formal and informal economies. During the program implementation period from 2019 to 2024, over 2,312 capacity-building trainings were conducted for over 38,331 SMEs and NGO representatives (including components 1, 2 and 3).
3. Consultancy and coaching services to SMEs on governance practice: The program provides tailored one-on-one consultancy and coaching services to SMEs on corporate governance codes. BEST project officers support SMEs throughout the implementation of the agreed-upon corporate governance actions, which include establishing formal structures and processes, strengthening internal controls, minimizing the probability of fraud losses, improving the level of transparency and disclosure, defining shareholder rights, and empowering the board of directors.
4. Training and workshops on transparency and anti-corruption for government officials: The BEST program, in cooperation with the Prevention and Public Awareness Department of the Independent Authority Against Corruption (IAAC), conducts training and workshops on transparency, anti-corruption, accountability, and informed policy-making decisions for aimag and soum-level governor office management staff. Over 2,937 government officials received BEST-supported anti-corruption training in the evaluation period.
5. Conducting SME development-related surveys: BEST conducts quantitative and qualitative surveys and empirical studies, which aim to assist policymakers, program partners, and private sector actors to understand the dynamics of the SME lending environment and to practice evidence-based decision-making and strategic learning. The analysis of this data is published online and shared with policymakers and business associations to improve the dialogue on private sector policy with quantitative evidence. The program published 16 surveys as of June 2024.
6. Provision of small grants: BEST implemented a small grant program starting in Year 2 to benefit disadvantaged small enterprises, start-ups, and NGOs determined to have exceptional merit and a likelihood of success if awarded a small grant. During the program implementation, BEST awarded 176 rural entities and start-ups received grants with a total value of MNT 1,714,569,000 (US$507,065) in small grants. 

The evaluation shall cover the following thirteen (13) program locations where the program has been implemented since its inception: (i) Ulaanbaatar, (ii) Darkhan, (iii) Orkhon, (iv) Selenge, (v) Dornogovi, (vi) Southgobi, (vii) Zavkhan, (viii) Khovd, and (ix) Bayankhongor, (x) Uvurkhangai (xi) Dornod, (xii) Khentii, and (xiii) Khuvsgul aimags. Four of these aimags, Orkhon, Dornogovi, Dornod and Khovd, are designated as regional hub centers.

IV. EVALUATION QUESTIONS

The evaluation will answer the following questions to assess the BEST program’s immediate outcomes and overall performance in Ulaanbaatar and 12 provinces of Mongolia, covering the period from December 2, 2019, to June 30, 2024.

1. To what extent were the assumed causal relationships between activities, outputs, outcomes and impacts in the Theory of Change supported by evidence?
2. To what extent were the BEST program’s intended outcomes likely achieved in Mongolia’s private sector/SMEs in the selected regions?

• To what extent were the program’s planned activities and strategies accomplished against the set target indicators?
• What were the unintended results/outcomes of the program in the target audiences, and to what extent were these outcomes likely to contribute to the overall effectiveness?

3. How has the BEST program navigated and adapted to unforeseen challenges and the evolving needs of SMEs?

• How effective were the adjustments made to the program’s activities during the Covid-19 pandemic in achieving its objectives?

     4. How effectively have the program activities and approaches addressed gender differences and gaps?

• How have the needs of women-owned SMEs been identified and integrated into the program’s activities?

     5. To what extent have the BEST program outcomes been sustained, or were likely to be sustained, even after the program ends?

• What approaches or strategies have been implemented to ensure the sustainability of these outcomes?

V.  EVALUATION DESIGN AND METHODOLOGY

This evaluation will be categorized as the final performance evaluation. It will focus on the BEST program’s performance and key results and will be guided by BEST’s theory of change (ToC) designed to lay the foundation of its local framework. Specifically, and based on the ToC, it will trace output/outcome relationships between the five (5) program components.

Triangulation should be used to gather data through a combination of techniques, namely: (1) desk review of secondary sources and other data found in reports generated through the program’s regular performance monitoring and internal evaluations, if any; (2) primary data collection through questionnaire surveys, key informant interview (KII), and focus group discussion (FGD), if deemed appropriate; and (3) direct and/or field observations by the evaluation team, among others. During the course of the evaluation, DS estimates the conduct of at least three (3) field visits to the rural aimags as well as Ulaanbaatar (UB) with a coverage of all five components. However, the evaluation team is encouraged to recommend other appropriate approaches and methods to answer the above evaluation questions.

The BEST program expects the analysis to consider gender and cross-cutting issues, such as the constraints to effective participation by persons with disabilities (PWD) and women and the opportunities to maximize effective participation of them in future program activities. Overall, the evaluation team should consider incorporating the use of gender-sensitive data collection methods and analysis of sex-specific differential outcomes throughout the evaluation.

The Evaluator will have access to all performance data, M&E database, surveys, and other deliverables that have been collected and reported by the program team as part of their Performance Management Plan for BEST. The evaluator must also review the following documents in preparation for the performance evaluation.

• Program proposals
• Annual and life of project implementation plans, Sustainability Plan, Activity Monitoring, Evaluation and Learning Plan, and Performance Management Plan
• Baseline survey report
• Program’s midterm evaluation report
• Relevant survey and research reports.

VI. DELIVERABLES AND REPORTING REQUIREMENTS

The following are the deliverables under this evaluation contract:

1. In-briefing Presentation: Within five (5) working days of contract commencement, the evaluation team will organize a briefing and workshop with DS and USAID/Mongolia for introductions to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and to finalize the evaluation approach and implementation plan.
2. Summary Report of Desk Study, Evaluation Approach, and Implementation Plan: Within two (2) calendar weeks of contract commencement, a 1) summary of desk study, 2) draft evaluation approach, and 3) implementation plan shall be completed by the Lead Evaluator and submitted to the Development Solutions. The summary desk study will include pre-identified documents and materials to be employed in the evaluation reference. The evaluation approach/design will include (i) a detailed evaluation design matrix (including the key questions, methods, and data sources used to address each question and the data analysis plan for each question); (ii) draft questionnaires and other data collection instruments or their main features; (iii) the list of potential interviewees and sites to be visited; and (iv) known limitations to the evaluation design. The implementation plan will include (i) the anticipated schedule and logistical arrangements and (ii) a list of the members of the evaluation team, delineated by roles and responsibilities.
3. Fieldwork Itinerary. The evaluation team will develop a draft itinerary for the fieldwork and submit it to DS for approval within three (3) calendar weeks after approval of contract commencement. The itinerary report will provide the fieldwork plan and logistical arrangements to be exercised together with an approved list of key informants and survey respondents.
4. Fieldwork Update. The fieldwork update report should provide updates on the overall data collection status and data assurance mechanism undertaken by the evaluation team, highlighting any challenges and measures undertaken during the exercise. As indicated in the previous section, DS estimates at least three (3) field visits to the rural sites and UB fieldwork to be conducted.
5. Draft Evaluation Report. The draft evaluation report should be consistent with the guidance provided in Section IX. The draft evaluation report will be a comprehensive report of all deliverables associated with the evaluation. The inception workshop will discuss specific data and reporting formats and other requirements. It shall not exceed 45 pages with no more than five (5) pages of executive summary, excluding annexes. The report shall follow USAID’s general guidance on Preparing Evaluation Reports. The first draft of the evaluation report shall be submitted at the latest nine (9) calendar weeks after contract commencement.
6. Out-briefing Presentation. The evaluation team will provide a debriefing presentation to DS and USAID/Mongolia consisting of a) a 5-10 pages summary of key findings, conclusions, and recommendations, and b) a PowerPoint presentation covering the above material. The draft debriefing presentation slides and the written summary will be submitted to DS for review and comment the week prior to the scheduled out-briefing date. These shall be finalized by the evaluation team for approval by DS. The out-briefing presentation shall be held at least five (5) calendar days prior to submitting the draft evaluation report.
7. Final Evaluation Report. The final report will be submitted in MS Word and PDF formats with a separate summary of not more than 10 pages. The report and summary will be in English and Mongolian and submitted within five (5) working days of receiving DS’ comments on the draft report.
8. Final publishable evaluation report. The report must be submitted to DS by October 15, 2024. Submission is inclusive of the following:

             a. Three (3) hard copies of the final report in English and Mongolian

             b. Three (3) USB flash drives containing:

• Electronic copies of the report (in PDF and MS Word formats)
• Supporting documentation inclusive of complete data collected
• Pictures and other visual materials, and
• Presentation materials.

VII. EVALUATION TEAM COMPOSITION

An enterprise that has been operating +5 years in the field of socio-economic, SME sector, international project/program research and evaluation (company, NGO, etc. State registration certificate must be attached)

An evaluation team shall carry out the final performance evaluation. It shall comprise a senior evaluation specialist, who shall act as team leader, and four team members—three evaluation specialists and one technical associate.

Team Leader should possess the following qualifications:

• At least a Master’s degree in evaluation and measurement, public/business administration, economics, development, management, and /or a relevant field.
• At least 10 years of experience conducting high-quality evaluations of development assistance programs/projects. A minimum of five evaluation reports, evaluation-related publications, journal articles, and similar documents under his/her senior leadership/authorship shall be submitted for review by DS.
• Adequate knowledge and skill in appropriate research and survey methods for conducting performance and impact evaluation. Adept in both quantitative and qualitative analyses and the use of appropriate software for data analysis.
• Strong technical writing skills and ability to adopt USAID technical and style guidance in report writing.
• Familiarity with USAID’s activity planning, implementation, monitoring, and evaluation, as well as the theory of change/development hypothesis approach to evaluating project/program outcomes, is preferred.
• Fluent proficiency in English is required.

Evaluation Specialist should possess the following qualifications:

• At least a Bachelor’s degree in evaluation and measurement, public/business administration, economics, development, management, and /or a relevant field.
• At least 7 years of work experience in evaluation and measurement and/or conducting evaluations.
• Proven experience in working with SMEs and related public and private institutional-level stakeholders
• Adequate knowledge and skill in appropriate research and survey methods for conducting evaluation. Adept in both quantitative and qualitative analyses and the use of appropriate software for data analysis.
• Fluent proficiency in English is required.

Technical Associate should possess the following qualifications:

• At least a Bachelor’s degree in business, management, accounting and administration and/or related field.
• At least 3 to 4 years of work experience in business management and administrative positions in research, evaluation and/or private sector.
• Adequate experience in coordinating logistical and financial backstopping for the evaluation
• Fluency in Mongolian and working knowledge of English is required

All evaluation team members should reside in the country upon commencement of the evaluation and throughout the evaluation process. No evaluation team member should have any past or present involvement with the BEST program. All team members must provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest.

DS and USAID/Mongolia shall review and approve the evaluation team composition. The Offeror may recommend an alternative team composition or structure that meets the SOW’s requirements better.

VIII.  EVALUATION SCHEDULE

Estimated Start Date: on or about July 29, 2024

Estimated End Date: on or about October 15, 2024

The team leader’s total level of effort (LOE) of up to 57 work days shall be spread over three months (from the date of contract awarding to the submission of the final report).

The table below shows the indicative tasks and timelines for the evaluation, excluding the weekends. The offeror shall refine this when s/he submits the evaluation design and work plan cited in Section VI.

Illustrative Schedule of Evaluation

IX. FINAL REPORT FORMAT

 The evaluation final report should include an executive summary, introduction, background of the local context and the activity being evaluated, the main evaluation questions, the methodology or methodologies, limitations to the evaluation, findings, conclusions, and recommendations, and lessons learned (if applicable), as described here. The report should be formatted according to the evaluation report template.

The executive summary should be 3–5 pages long and summarize the purpose, the program’s background, main evaluation questions, methods, findings, conclusions, recommendations, and lessons learned (if applicable).

The report shall explain the evaluation methodology in detail and disclose limitations to it, with particular attention to limitations associated with it (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.).

The evaluation’s primary language is English, and it includes reports, related data, and instruments. The final evaluation report and associated data (evaluation questions and findings) shall also be in Mongolian.

The annexes to the report shall include:

• The evaluation SOW
• Any statements of difference regarding significant unresolved differences of opinion by implementers, participants, stakeholders, and/or members of the evaluation team
• All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides
• Sources of information, properly identified and listed, and
• Disclosure of conflict-of-interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest.

X.  CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT

Per the USAID Evaluation Policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.

• The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the program, what did not, and why.
• Evaluation report should adhere to plain language guidelines per the USAID Style Guide.
• Evaluation report shall address all evaluation questions included in the SOW.
• Evaluation should be based on the best methods of appropriate rigor. Evaluations must produce well documented findings that are verifiable, reproducible, and on which stakeholders can confidently rely, while providing clear explanations of limitations. Evaluation methodology should be explained in detail and sources of information should be properly identified. Sufficient information on methodology and data collection should be included to allow stakeholders to make informed judgments about the quality and accuracy of the findings, and to allow other evaluators to replicate the protocol.
• Evaluations should be independent, objective, and unbiased in measuring and reporting; limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluators should strive for objectivity in the planning and conduct of evaluations and in the interpretation and dissemination of findings, avoiding conflicts of interest, bias, and other partiality.
• Evaluation report should adequately capture the situations and experiences of people of different genders. If evaluations findings or data include people-level indicators, they must be disaggregated by sex, while considering the contributions toward local and urban areas.
• Findings, conclusions, and recommendations (if any) should be specific, concise, and supported by solid quantitative or qualitative evidence.
• Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions.
• Conclusions should be based on the evaluation findings.
• If recommendations are included, they should be supported by a specific set of referenced findings, and should be prioritized, action-oriented, practical, and specific. Evaluations should be oriented to reinforcing local ownership; when possible, evaluators should include relevant local stakeholders in joint development of recommendations.

XI. OTHER REQUIREMENTS

All quantitative data collected by the evaluation team must be provided in an electronic file in an easily readable format agreed upon with DS. The data should be organized and thoroughly documented for those unfamiliar with the program/project or the evaluation. DS and USAID/Mongolia will retain ownership of the review and all datasets that have been developed.

ANNEX I. SUBMISSION OF THE OFFER         

Interested entities are invited to submit the following documents:

Technical Proposal [not exceeding 30 pages]

• Organizational background showing evidence of prior performance in conducting evaluation for development projects.
• Outline of proposed services, quality controls, and proposed data collection methods, tools, and analysis.
• Implementation timetable to carry out the tasks under Section VIII
• Offeror’s technical assurance to carry out the quantitative and qualitative surveys-proven data collection platforms and tools such as tablets etc shall be attached
• List of proposed Evaluation Team Members and Curriculum Vitae (CV) for each member. (Each CV must not exceed 5 pages).
• At least three (3) references for successful prior projects of a similar nature. These references should include contact names, e-mail addresses, and telephone numbers of people who can be contacted regarding the Offeror’s prior performance.

Cost Proposal

• The estimated cost for evaluation is MNT 135,000,000. The Cost Proposal should include estimated costs (MNT) per main activity throughout the evaluation. The suggested template (which can be modified if necessary) is available in Excel format. It should specify the detailed cost breakdown and the total price of the services offered in response to this SOW.

The Offeror is suggested to submit via procurement@dsmongolia.org and purevdulam@dsmongolia.org with the title of Final Performance Evaluation Offer for BEST Program before or at 17:00, July 15 (Monday), 2024.

Questions/Queries regarding this SOW should be delivered to the task manager: Mrs. Purevdulam Jamiyansuren – purevdulam@dsmongolia.org +976 95195331 or 88089553

ANNEX II. TECHNICAL EVALUATION CRITERIA

The interested entity’s technical proposal will be evaluated based on the following criteria:

1. Technical Proposal (evaluation weight: 40 points)

• The degree to which the proposal incorporates a sound understanding of the evaluation requirements is elaborated in Section V (Evaluation Design and Methodology) of the Statement of Work. (10 points)
• Proposed overall methodology, including sampling and data collection methods (10 points)
• Schedule for delivery of evaluation deliverables (10 points)
• Quality assurance and control methods regarding overall evaluation task, including field data collection, data analysis and synthesis and reporting (10 points)

     2. Personnel qualification and expertise (evaluation weight: 60 points) 

• Past Experience evaluating development projects using multiple approaches, especially in the private sector and SME development. Previous experience in a similar context of performance evaluation and working experience with USAID requirements/formats are advantages. (30 points)
• Relevant academic backgrounds/degrees of the evaluation team (15 points)
• Knowledge of challenges facing SMEs with an emphasis on rural SMEs (15 points)

No rating is assigned to the financial offer. While the technical offer criteria are significantly more important than cost, cost remains essential. This evaluation will consist of a review of the cost proportion of a proposal to determine if the overall costs proposed are realistic for the work to be performed, if the costs reflect the Offeror’s understanding of the requirements, and if the costs are consistent with the technical offer. While the Offeror’s qualifications are substantially more important than costs, cost may become a deciding factor in the award.

ANNEX III. PROPOSED PAYMENT SCHEDULE, PAYMENT TERMS & DELIVERABLES

Based on successful completion and subject to acceptance of all work and services, including the submission of required reports and deliverables, payment shall be as follows:

ANNEX IV. ESTIMATED LOE IN DAYS BY ACTIVITY FOR TEAM

[1] Please note that the program results documented here are current as of June 15, 2024.