Unit One – Program Design, Implementation and Evaluation
Why should Advanced Master Gardeners study program design, implementation and evaluation?
Advanced Master Gardeners are expected to manage all aspects of comprehensive, long-term programs that will result in measurable impacts that meet a specific community need. The approach is the same used by other non-profit organizations. Every detail is important, or goals will not be met and support and funding may disappear. If not planned properly, gaps or flaws may result in needs not met and potentially embarrassing and financial setbacks for the program.
U.S. Environmental Protection Agency, Office of Inspector General. 2012. “Evaluation of Costs Claimed Under EPA Cooperative Agreements CB-97324710 Through CB-97324705 Awarded to Alliance for the Chesapeake Bay, Inc.” Accessed April 9, 2014. http://www.epa.gov/oig/reports/2012/20120822-12-4-0720_glance.pdf.
Keene, Courtney. 2007. “Case Studies of a few ‘Failed’ Projects.” Selections reprinted with permission. Accessed April 14, 2014. http://www.globalhood.org/casestudies.shtml.
Blumstein, D.T. and C. Saylan. 2007. “The Failure of Environmental Education (and How We Can Fix It).” PLOS Biology 5(5): e120. Accessed March 22, 2014. http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.0050120.
Role of the Advanced Master Gardener Steward as Program Manager
The Advanced Master Gardener program was created as a growth opportunity for certified Master Gardeners and to extend the ability of the local unit to address community needs. Provided to each student is the Program Design, Implementation and Evaluation Manual, which describes the goals of the program, requirements and resource materials.
The Advanced Master Gardener must be aware that their work is to be part of a program and not disjointed projects. The word “program” is used in many ways – a single education event, the agenda for a meeting, etc. For our purposes, the definitions are: “program”– a plan of things that are done in order to achieve a specific result – and “project” – a planned piece of work that has a specific purpose (such as to find information or to make something new) and that usually requires a lot of time. Because of their differences, programs and projects necessarily involve different responsibilities.
Reilly, Patricia M., editor. 2014. Program Design, Implementation and Evaluation, A Manual for the Advanced Master Gardener. Second Edition. Blacksburg, VA: Virginia Tech.
Chan, Kimberley. 2013. “Project vs. Program Management.” Reprinted with permission. Accessed July 3, 2014. http://www.onedesk.com/2013/06/project-vs-program-management.
Need Assessment before Program Idea Generation
When learning new concepts and solutions as Advanced Master Gardeners do in their technical training, excitement often pre-empts planning. Planning is difficult for some, is time consuming and is frequently given little time in preference for initiating the activity. However, it is necessary to ensure success for the program.
Many fields use a needs assessment before launching new programs, and when the topic is addressed fully, books are filled. In a time when resources are limited, Cooperative Extension promotes the use of needs assessments to make informed decisions on programs of strategic priority that will meet community needs.
McCawley, Paul F. 2009. “What is a Needs Assessment?” In Methods for Conducting an Educational Needs Assessment. University of Idaho Extension, 1-7. Accessed April 14, 2014. http://www.cals.uidaho.edu/edcomm/pdf/bul/bul0870.pdf.
Stakeholders, Customers, and Target Populations
It is easy to think of program participants as the ones who our programs directly impact – the ones who ask questions at clinics, attend our classes or enroll their lawns in turf programs. A successful Advanced Master Gardener program will have impact on more than just the people who attend a class or workshop – it will have value to the larger community. The challenge is to brainstorm all the possible individuals and groups that may have a stake in the issue and utilize their collective knowledge in determining needed programs.
U.S. Environmental Protection Agency, Office of Water, Nonpoint Source Control Branch. 2013. “Getting In Step: Engaging Stakeholders in Your Watershed:” 3-10. Accessed April 16, 2014. http://cfpub.epa.gov/npstbx/files/stakeholderguide.pdf.
Gibson, Terry L. 2001. “Key Informant Approach.” In Cooperative Extension Program Planning in Wisconsin, Terry L. Gibson, editor, 39-41. Madison, Wisconsin: University of Wisconsin-Extension. Accessed December 5, 2014. http://www.uwex.edu/ces/pdande/planning/pdf/ProgramPlanning.pdf.
Figuring out how to change behaviors is key to constructing a program that will attain the results desired from an Advanced Master Gardener – Water Manager program. Research, focus groups, surveys and key informant interviews can lead to identifying perceived barriers to adopting desired behaviors. Community-based social marketing suggests a process that has had success in changing behaviors. The “fishbone” or cause-and-effect diagram is used to discover probable reasons for a behavior, leading to hypotheses of the root cause which can then be tested through research and discussions with stakeholders.
McKenzie-Mohr, Doug. 2000. “Promoting Sustainable Behavior: An Introduction to Community-Based Social Marketing.” Journal of Social Issues 56:543-538. Accessed April 14, 2014. https://web.stanford.edu/~kcarmel/CC_BehavChange_Course/readings/Mckenzie_socialmarket_2000.pdf.
Watersheds Stewards Academy. No date. “Fertilizer.” Accessed August 22, 2016. http://www.aawsa.org/fertilizer.
American Society for Quality. No date. “Fishbone (Ishakawa) Diagram.” Accessed April 9, 2014. http://asq.org/learn-about-quality/cause-analysis-tools/overview/fishbone.html.
Program Theory, Logic Model and Service Utilization Plan
You must have a sound theory for why the activities that you plan for your program will affect the condition or behavior that the program is to change or improve. Program theory, if incorrect, can waste resources and not result in the desired outcomes. Logic models visually represent program ideas and are useful for presenting to decision-makers the current situation, what resources are needed to conduct the program, how it will work and what outcomes can be expected.
Amherst H. Wilder Foundation. 2009. “Program theory and logic models.” Reprinted with permission. Accessed July 3, 2014. http://www.evaluatod.org/resources/evaluation-guides/LogicModel_8-09.pdf.
Israel, Glenn D. 2001. “Using Logic Models for Program Development.” Reviewed 2013. Reprinted with permission. Accessed April 21, 2014. http://edis.ifas.ufl.edu/pdffiles/WC/WC04100.pdf.
University of Wisconsin Extension, Program Development and Evaluation. Logic Model Example: Water Quality Program. Reprinted with permission. Accessed August 22, 2016. https://fyi.uwex.edu/programdevelopment/files/2016/03/WaterQuality.pdf.
Simply answering questions or conducting a class for program participants is not a sure way to change behaviors. Much research has been done on the psychology behind changing behaviors, and several theories exist. Community-based social marketing is not only useful for determining reasons for a behavior but the findings can be used to create plans to change behaviors. Prochaska’s and DiClemente’s Stages of Change Model provides the characteristics and techniques associated with each stage of change. The model importantly provides time frames so that expectations of the Advanced MG Water program may be accurately set.
University of Arizona, Step Up! Program. No date. “Prochaska and DiClemente’s Stages of Change Model.” Reprinted with permission. Accessed September 19, 2014. http://www.stepupprogram.org/docs/handouts/STEPUP_Stages_of_Change.pdf.
Hopwood, Toby and Rowena Merritt. 2011. Big Pocket Guide to using social marketing for behavior change. London: National Social Marketing Centre. Accessed September 7, 2014. http://www.thensmc.com/sites/default/files/Big_pocket_guide_2011.pdf.
URS Australia. 2007. “Promoting Behavioural Change in Household Water Consumption: Literature Review.” Report prepared for Smart Water, University of Victoria, Australia, 2-1 – 2-1.7. Accessed August 22, 2016. http://www.vu.edu.au/sites/default/files/Promoting%20behavioural%20Change%20in%20Household%20Water%20Consumption.pdf.
Customers and Quality Characteristics
Responding to customer needs is a critical component of the relationship between the Advanced Master Gardener – Water Program Manager and the customer. Meeting the needs of the customer is an apparent requirements, but the program must also identify the specific characteristics wanted or needed by the customer.
Rudnick-Kaun, Mary Lee. 2002. “Quality Basics: Customer Requirements and Specifications.” Wisc-Online, Wisconsin’s Technical Colleges. Accessed September 17, 2014. http://www.wisc-online.com/objects/ViewObject.aspx?ID=QLT702.
Understanding how to develop good performance measures that accurately guide activity creation and describe results of your program is important in the planning stages as well as during evaluation. Measures must be relevant, and aligning them to the logic model will help. Too often claims are made about accomplishments of programs; the Advanced MG must avoid such statements.
U.S. Environmental Protection Agency. 2016. “Evaluating EPA’s Programs: Program Evaluation and Performance Measurement at the EPA.” Accessed August 22, 2016. https://www.epa.gov/evaluate/program-evaluation-and-performance-measurement-epa#performacemaeasurement.
Bortniak, John. 2012. “How to Write Great Performance Measures.” Presentation for the U.S. Department of Commerce, National Oceanic and Atmospheric Administration, November 5. Accessed December 21, 2014. http://www.lib.noaa.gov/about/news/PerformanceMeasures101Tutorial.pdf.
Virginia Department of Planning and Budget. 2006. “Refining Agency Measures: Performance Management Workshop.” Accessed September 17, 2014. http://www.dpb.virginia.gov/forms/20090724-5%5CPerfMeasureWorkbook.pdf.
Reporting Results, Impact Statements and Public Value
Data collection, analysis and reporting of findings are an essential part of the Advanced Master Gardener program. As such, careful planning is critical for ease of data collection and timing of reports. Analysis of data must be without bias and conclusions must be drawn accurately.
Virginia Cooperative Extension has adopted formats for impact reporting. Learning opportunities and templates exist for relaying the public value of programs.
Rabinowitz, Phil and Stephen B. Fawcett, contributors. 2014. “Collecting and Analyzing Data.” In The Community Tool Box. University of Kansas. Accessed December 5. http://ctb.ku.edu/en/table-of-contents/evaluate/evaluate-community-interventions/collect-analyze-data/main.
Virginia Tech, College of Agriculture and Life Sciences. 2014. “Writing Effective Impact Statements: Who Cares? So What?” Accessed August 22, 2016. http://www.communications.cals.vt.edu/resources/impact-statements.html.
University of Maine Cooperative Extension. No date. “Public Value – Public Value Statements for Master Gardener Programs.” Accessed August 22, 2016. https://extension.umaine.edu/community/public-value/statements-for-master-gardener-programs.
Program Evaluation and Continuous Improvement
Planning for, and evaluating the Advanced Master Gardener Water Manager’s program may be one of the most important activities that the Advanced MG conducts. Evaluations are opportunities to systematically review and gauge the value of an entire program or can be as simple as evaluating one element or activity of the program. While evaluations are each unique, step-by-step instructions are and examples available to guide the Advanced MG.
Milstein, Bobby, Scott Wetterhall and the CDC Evaluation Working Group, contributors. 2014. “A Framework for Evaluation.” In The Community Tool Box. University of Kansas. Accessed December 5, 2014. http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/framework-for-evaluation/main.
Virginia Cooperative Extension. No date. “VCE Program Evaluation and Reporting.” Accessed September 14, 2014. http://www.intra.ext.vt.edu/support/eval.html.
Boyd, Heather. 2002. “Ways to Improve the Quality of Your Program Evaluations.” Public domain. Accessed September 8, 2016. https://fyi.uwex.edu/programdevelopment/files/2016/04/Tipsheet9.pdf.
U.S. Environmental Protection Agency, Office of Policy. 2013. “Evaluation of the Role of Public Outreach and Stakeholder Engagement in Stormwater Funding Decisions in New England: Lessons from Communities,” Executive Summary, ii – viii. Accessed August 22, 2016. https://www.epa.gov/sites/production/files/2015-09/documents/eval-sw-funding-new-england.pdf.
American Society for Quality. No date. “Continuous Improvement.” Accessed August 22, 2016. http://asq.org/learn-about-quality/continuous-improvement/overview/overview.html.
Evaluation Skill Practice
Three VCE program publications report on various programs and are presented here for the purpose of practicing evaluation skills. The reports were not written to serve as evaluations. Select one or more of the programs, review the reports as if you were the external evaluator of the program. Try to answer the following questions:
Who is the audience for this report?
In what stage of development is this program? Given its maturity, what approach could the evaluation take?
Who are the stakeholders and who is the target audience?
What need is there for the program?
What level of programming does this program seek to achieve?
What are the expected outcomes – short-term, intermediate, and long-term?
Describe the program’s activities and define the link between the activities and expected outcomes.
What resources are expended on the program?
Has the program been successful in achieving goals or meeting performance measures/
What kinds of advice would you offer as opportunities for the program managers to improve how the program and accomplishments are communicated?
Smith, Robert. 2010. “Renewable Resources Extension Act (RREA) Programs.” Reprinted with permission.
Virginia Cooperative Extension – Madison County. 2012. “Northern District 2012, Forestry & Natural Resources Selected Program Highlights.” Reprinted with permission.
Crall, Alycia W. 2013. “Virginia Master Naturalist Program.” Reprinted with permission.
American Evaluation Association – http://www.eval.org.
Rossi, Peter H., Mark W. Lipsey and Howard E. Freeman. 2004. Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage Publications.
Shackman, Gene. 2012. “Free Resources for Program Evaluation and Social Research Methods.” Accessed September 14, 2014. http://gsociology.icaap.org/methods/.
Volunteer Development Resources – http://www.intra.ext.vt.edu/support/volunteer/index.html.