Appendix III: Evaluation Worksheet
The evaluation worksheet OTF staff and our Advisory Council use to evaluate proposals.
The below worksheet is used by both OTF staff and our Advisory Council to evaluate proposals. The scoring system for each area is on a 1-5 scale, with 1 being ‘poor’ or ‘very weak,’ 5 being ‘excellent’ or ‘very strong,’ and 0 being non-applicable. In general, those with higher aggregate scores will make it farther in the process.
The below is asking you for those general thoughts you have after reviewing a proposal. It is a space to include comments and questions for areas we do not specifically ask for a rating. Comments and questions posted here will be aggregated back to the Advisory Council and to the proposal authors.
Provide here any specific bits that got you really excited or that you see as positive about this proposal.
Provide here any specific bits that leave you feeling uneasy or that you feel warrant concern regarding this proposal.
Provide here anything you found that warrants dire warnings, notice of very serious dangers, and/or potential disasters.
This section is asking you to score and comment on specific proposal topics. Comments and questions posted here will be aggregated back to the Advisory Council and to the proposal authors.
Scoring system: 1 is ‘poor’ or ‘very weak’ and 5 is ‘excellent’ or ‘very strong’, 0 denotes more information is needed – see questions for proposal author –, and n/a for sections the reviewer does not want to rate.
Considerations on project overview: Are the project’s goals clear? Are the projects goals realistically met by the proposed solution? Does the proposal identify and acknowledge what the challenges will be? Does the proposal state how it is done today and the limitations? Is it clear and specific who could benefit and what will the impact be if the project is successful? Does the proposal cite an actual use case? Does the proposal state how much it will cost and how long will it take?
Considerations on objectives: Does the proposal state a clear set of objectives and tasks for the proposed effort?
Considerations on Methods and Strategy: Does the project propose methods appropriate to the goals and objectives? Does the project effectively apply its stated methods? Does the proposal suggest modified procedures in response to changing circumstances? Will the project fill a potential need or function that is currently unfilled, be reinventing the wheel or creating a solution in search of a problem? Is the proposed solution viable in the real world?
Considerations for technical feasibility: Does the proposal clearly state the efforts technical goals? Are objectives articulated technically and succinctly? Does the proposal explain what is new in the approach and why it will succeed? Does the project increase or decrease known attack surfaces? Does it create new surfaces? Does the project buy tactical breathing space for existing problems or push towards convergence by changing the playing field? Does the project identify any expected or past hurdles in achieving technical goals?
Considerations for red teams: Does the proposal identify or recognize potential incentives to an adversary? Does the proposal consider potential illicit uses of the project? Does it discuss how an adversary might use the solution to further their own goals? Does the proposal identify potential unintended consequences of the project? Does the proposal identify and understand the proposed projects asymmetry? Given an attacker and defender of the project, does the proposal explain which role is more advantageous (effort, cost, time, etc.) and why? Does the proposal explain medium and long‐term strategies from both the attacker and defender point of view. Does the proposal discuss how to defeat the project, identify its deficiencies, or does it presume there are none?
Considerations for usability: Does the project demonstrate a high degree of usability/accessibility? Does the proposal demonstrate external demand i.e., demand originated from potential users rather than from would-be patrons of some possibly hypothetical set of users? Does the projects focus impact either a small number of high value or at-risk users or a more general large numbers of users?
Considerations for sustainability: What is the project’s plan for future development/implementation? Does the proposal state if the project has other funding sources? Does the project currently receive any U.S. government funding? Does the project identify any cost sharing or matching for the proposed effort? Will the project be able to support itself by the requested funding, community sources, or other in-kind or indirect support? Does the project have a diversified funding/support stream i.e., how dependent would the project be on OTF?
Considerations for collaborations: Does or should the project support a collaborative open source community? How does the project facilitate inter-project collaboration, including: talking with other projects doing similar things and identifying potential points of overlap; acted/planned to modularize code to enable others to reuse? Do the proposed deliverables assist other OTF projects/goals beyond this one project? Do the proposed deliverables assist other Internet freedom projects/goals beyond this one project?
Considerations for rationale: Is the proposed project and its trajectory in-line with OTF’s principles and goals?
OTF's mission
The Open Technology Fund (OTF) is a next-generation program by Radio Free Asia (RFA) that uses public funds to support open technologies and communities that increase free expression, circumvent censorship, and obstruct repressive surveillance as a way to promote human rights and open societies.
Guiding principles
Freedom of speech and expression, freedom of the press, open exchange of ideas and information, open Internet, Internet freedom, forward-thinking ideas and innovation, open philanthropy, transparency, accountability, alternative methodologies, new technologies, and collaboration
Program focus areas
Research in how Internet interference on modern communication networks occurs and to discover the technologies and methodologies that can circumvent interference; Development of the technologies required to circumvent censorship and increase communication safety; and, Implementation of circumvention tools for widespread use and adoption amongst non-technical citizens affected by censorship, interference, and illegitimate surveillance.
Considerations on cost realism: Is the budget realistic and commensurate with both the project needs and time frame? Is the project going to require funds immediately upon proposal approval?
Considerations for qualifications: Are project team member(s) clearly identified, along with work experience, in the proposal? Does the project’s team posses the skills uniquely qualifying them to complete the proposed scope of work? Does the project team have a history of successful work related to the current initiative? Does the project have a core team (leadership, developers, etc.) dedicated to this project? Have team members worked with at-risk communities in the past?
Considerations for the ability to evaluate the project: Does the project articulate a measurable set of evaluation criteria and milestone metrics against results? Are the metrics both quantitative and qualitative? How difficult will an assessment of success or failure be?
Do you recommend supporting this proposal’s statement of work for the requested amount?
(space for any additional comments not captured in or suitable for the above sections)