For review in advance of September 18 Civic Alliance meeting.

DRAFT CONCEPTS for a LIFE CYCLE PERFORMANCE SCORECARD
For Rebuilding Lower Manhattan and Related Regional Revitalization

By Paul D. Epstein, Epstein and Fass Associates, and the American Society for Public Administration: CAP and NY Metro Chapter

INTRODUCTION AND RATIONALE
Civic planning and public engagement processes of the Civic Alliance and other groups may continue to influence public officials and private developers concerning plans made for rebuilding Lower Manhattan. However, those plans, and all the civic effort expended to influence them, will be for naught if plans are not adequately implemented and desired outcomes not attained. Some kind of objective accountability tool is needed to track progress in planning, funding, and implementing important initiatives for rebuilding and achieving desired measurable public outcomes, both downtown, and, where relevant, for the wider region (e.g., as in several Civic Alliance economic development and social justice recommendations). Indeed, if public officials—especially elected officials—know that a 'Scorecard' is in development based on recommendations of a broad array of civic groups, and that public 'scores' will be periodically reported and given media attention, those officials are more likely to include civic groups' recommended initiatives in their plans in the first place, and work to get them funded and implemented, in order to have higher performance scores reported. A performance scorecard can also be more than an accountability tool. It can be used to help promote a continuing civic dialog between civic groups and decisions makers as rebuilding progresses, to help civic groups and decision makers fine tune and improve rebuilding and revitalization strategies over time, and to provide a focus for continuing citizen engagement in the rebuilding process.

CONCEPTUAL STRUCTURE OF PROPOSED SCORECARD

Two Main Scorecard Dimensions: Performance and Time

Planning for rebuilding Lower Manhattan has been a dynamic process, and will continue to be. But even as plans become solidified, the rebuilding process itself will be dynamic. Many desired outcomes for the end results of rebuilding (e.g., popular acclaim and high use of the new urban spaces and memorial, balanced economic growth for all, residential growth and affordable housing, increased use of key mass transit modes, increased community activity around the clock, energy efficient and ecologically sound buildings) will not be realized for years. To be useful for keeping decision makers accountable, and keeping the rebuilding and revitalization process on track, a scorecard must view performance dynamically, tracking different kinds of results for most initiatives over time, such as the extent to which desired projects are funded, achieve various stages of implementation, and ultimately contribute to desired public outcomes.

To match the dynamism of rebuilding, the scorecard will have two major dimensions:

Performance related to specific goals, issues, and initiatives.
Time: 'Where we are' vs. 'Where we should be' at any point in time (each reporting period).

The meaning and expectations of 'performance' for each issue and initiative tracked, and thus of 'where we should be,' will change over time. However, the different aspects of performance can be planned in advance, creating a dynamic progress plan for tracking results through all stages of the life of each initiative. Thus this approach is a 'Life Cycle Performance Scorecard.'
'Layered Index' Approach to Develop 'Scores' and Communicate Performance
The performance scorecard will be developed in layers, representing different levels of detail, to facilitate communication of attention-getting summary 'scores' (e.g., for media and political attention) and of the complex data behind the scores (e.g., for citizen advocates wanting to evaluate particular issues in depth, for program and project managers wanting to determine how to improve performance and their 'scores'). The exact number of layers may vary by major issue, but the basic layering structure of the whole scorecard will follow this pattern:

Two distinct 'outer layers' of consolidated performance indices that yield 'scores':
The 'Total Score for Rebuilding and Revitalization Performance' based on a composite index of scores for about five to ten major issues or high-level goals.
The 'Issue Score' for each major issue or high-level goal, based on a subsidiary 'issue scorecard' or index for each major issue or goal.
The 'total score' and 'issue scores' would be the most widely distributed information, along with summary explanations and graphics, designed to communicate a quick, compelling grasp of how well rebuilding is going, and to capture widespread public and media attention.
More detailed 'inner layers' of actual performance data for each major issue or goal, and for the underlying initiatives and desired outcomes related to each major issue, will provide the detailed factual basis behind each 'issue scorecard,' and will provide layers of higher complexity that people interested in particular issues can dig into for evaluation, advocacy, policy assessment, and adjustment of projects and programs to improve future results.

Structuring the Scorecard Indices on the Emerging Civic Alliance Policy Matrix

To be credible, the Rebuilding Performance Scorecard should be anchored in the work that emerges from public and civic processes. The policy matrix being developed by the Civic Alliance can provide that credibility, because it will be based on the recommendations of several major civic planning processes organized by various organizations. Assuming the policy matrix ends up fairly representing the contributions of the various civic groups, then basing the scorecard structure on the structure of the policy matrix will lend the credibility of this broad range of civic groups to the scorecard. As of August 20th, the draft policy matrix was structured into eight major issue groups. Thus, one way to structure the Rebuilding Scorecard can be to create eight 'issue scorecards'—one for each major issue group in the policy matrix—and assign each so many ' percentage points' on the overall scorecard, the sum of which equal 100%. To limit the level of complexity of this paper, illustrative examples of the 'outermost layer' of a Rebuilding Performance Scorecard simply use the same eight major issue groups as the draft policy matrix, and assign each of them an equal weight of 12.5 percent of the 'total score.' Figure 1 shows how at any given time, the total score is built from the scores on each of the eight issues, which in turn are built from their own respective issue scorecards.

The eventual structure of the scorecard, while still based on the policy matrix, may end up more complex than a combination of eight equally-weighted issue scorecards. In one approach, some percentage points could be allocated to a set of 'overarching goals' that are thought to be achievable only if progress is made on multiple issues (e.g., 'create a vibrant, 24-hour mixed use community'). For example, a set of four or five overarching goals might be assigned a total of 20 points, and each of the eight issue scorecards assigned 10 points each, for a total maximum score of 100%. A set of overarching goals, or some other device, can be used to reflect the integrated nature of the major issues, and the need for synergistic responses to all the issues for effective revitalization. Also, as the policy matrix and scorecard are developed further, it may make sense to combine related issue groups that have fewer individual recommendations or initiatives, or to expand the definition of an issue group defined narrowly in the policy matrix. For example, as of August 20th, the 'Regulatory' issue group was narrowly defined only around SEQR. If it remains that narrowly defined, one possibility for the scorecard would be to move SEQR into 'Green Buildings/Environmental Concerns,' and not have a separate Regulatory issue scorecard. Alternatively, the Regulatory issue group could be expanded into a broader 'Public Process' or 'Governance' issue group, that not only covers SEQR and possibly other regulatory processes, but also includes keeping critical planning and decision making processes open, participatory, and transparent throughout the lifecycle of planning and rebuilding. Finally, the various issue groups do not have to be weighted equally in the total scorecard. Two suggestions for how unequal weights might be assigned to different issue scorecards are:

Develop a consensus among the civic groups represented in the policy matrix on what specific weights (or percentage points) should be allocated to each issue scorecard;
Use an open, participatory public process of some kind (e.g., part of a future 'Listening to the City' event) to develop the weights for each major issue group or issue scorecard.

PUBLIC COMMUNICATION OF RESULTS

Paper Reports
At each Scorecard reporting period (e.g., annual, every six months), a range of paper reports might be issued at various levels of detail. To encourage most recipients to read and consider the performance information, the most widely-distributed paper reports will be kept short, with data reported at a summary level, accompanied by concise explanatory graphics and text. The 'lead' for these reports can be, for example, the 'Total Rebuilding Score' and the two or three major reasons the score is at that level for that reporting period.

'Front Pages' of Printed Reports: A 'front page graphic' might be a simple double bar chart such as that shown in Figure 2, with one bar simply showing the total score out of a maximum of 100%, and the second bar depicting the 'issue score' for each major issue. The second bar would make it clear how much progress—or the lack of progress—in, say, transportation, housing, and economic development each contributed to the total score for that reporting period. Another possible comparison to include near the front of each printed report is the current total score vs. the total scores for the last several reporting periods, so readers can assess overall progress over time, perhaps with a trend-line graphic to illustrate the performance trend.

'Back Pages' of Some Printed Reports: Some printed reports may go no further than a graphically interesting summary front page, accompanied by brief descriptions of the major progress made on some issues, and key disappointments that led to low scores on other issues. Other printed reports may go further, with 'back pages' of perhaps 2 to 3 pages per issue group with a bar chart to illustrate each issue scorecard and some level of text explanation for the various scores. Trend lines of scores over time might also be included for each issue group.
Communicating Results on the Internet
The Internet provides a cost-effective opportunity to provide access to much more detailed data for self-customized inquiry and analysis by many individuals and groups based on their own particular interests. A Rebuilding Performance Scorecard Web Site could include a 'home page' or set of 'summary pages' very much like the 'front pages' and 'back pages' of the printed reports, perhaps with the same 'front page graphic.' However, the home page and summary pages would include links to much more detailed layers of data, providing a better idea of the complexity involved in assessing progress and results for each issue and initiative. These more detailed pages could include more detailed graphics and maps. This web-based layered, linked approach to reporting data would give users the ability to 'drill down' to levels of detail of interest to them about particular issues or initiatives. Depending upon the characteristics of data collected for some data elements, and software used, it could also be possible for users to customize their own analyses, maps, or graphics. The scorecard web site would also provide a full explanation for how the scorecard system works, all the weights, all the data sources, other technical information, and the civic groups who participated in developing the scorecard.

DEVELOPING AND USING THE TIME DIMENSION OF THE SCORECARD
Developing the 'time dimension' is critical to making the scorecard dynamic, in order to track progress and results through the various life cycles of the major initiatives to rebuild Lower Manhattan and revitalize the region. The time dimension will be used to determine 'Where we should be' for each major initiative related to each major issue for each reporting period. Then, a set of comparisons of 'Where we are' vs. 'Where we should be' for all the initiatives related to an issue will be used to determine the 'issue score' for that issue. It would work as follows:

For each key initiative or set of initiatives within each issue, a 'Planning and Implementation Timeline' would be developed that takes that initiative from being included in formal plans by a 'cognizant organization,' to being funded, to being implemented, and finally to yielding or contributing to measurable desired outcomes, as shown in Figure 3. (A 'cognizant organization' is a public or private organization with legal authority to plan, approve, fund, implement, or enable all or part of the initiative in question.)
Some--perhaps most--initiatives within particular issues may so large or complex to need to be planned, funded, and built or implemented in phases, so the timeline for those issues should reflect the expected progress of each phase of those initiatives, as noted in Figure 3.
There may not always be a one-to-one relationship between objective measures of desired outcomes and specific initiatives tracked for each major issue. Some initiatives may be expected to contribute to several desired outcomes, and some desired outcomes may be expected to result from several different initiatives.

ORGANIZING FOR, AND DEVELOPING SCORECARDS

The key to building an overall scorecard is to build credible 'issue scorecards' for each major issue. To develop an issue scorecard, the first step is to organize the appropriate players who, collectively, have the credibility, knowledge, and skills to develop a technically competent, practical, believable, and cost-effective scorecard for that issue. The next step is to develop the scorecard, preferably through a process that includes some level of public involvement.

Suggestions for Organizing the Players

People will need to be organized to develop each major Issue Scorecard, and to develop and sustain the overall Scorecard effort. One professional organization that may be a good source of people to assist this process overall, and on an issue-by-issue basis, is the American Society for Public Administration (ASPA). The New York Metro Chapter of ASPA is a member of the Civic Alliance, and many chapters members work in public agencies that have expertise and data sources for key issues of concern to the Alliance. ASPA has championed and supported public performance measurement nationally for many years, and has an active 'Center for Accountability and Performance' (CAP) of which the author of this paper is a founding director and current fellow. An advocacy organization in the Civic Alliance, the Straphangers Campaign, may also be a good resource for a Scorecard effort, even beyond the group's normal focus on public transit. The Straphangers have demonstrated experience developing index-based rating systems with some similarities to the system envisioned for the Rebuilding Lower Manhattan Scorecard, and they have proven especially good at getting media and public attention to their ratings and performance reports.

Organizing the players for each major Issue Scorecard:

Develop a cognizant 'civic oversight group' for each major issue. They may be drawn, for example, from among members of the groups involved in the policy matrix, or with particular measurement knowledge or expertise from other groups such as ASPA.
Invite cognizant public agencies to participate and provide support (e.g., data, staff support, meeting spaces).
Provide adequate supplementary research and process support, which may vary by major issue depending upon the capabilities and resources brought to the process by each member of the group and the level of support provided by public agencies.

Organizing the players for the overall Scorecard:

Form a small 'Core Scorecard Team' of people who bring appropriate process facilitation and technical knowledge and skills, and who can help bring key players together to build and support the overall Scorecard and its component parts.
Invite cognizant public agencies with broad rebuilding responsibilities and multi-issue concerns (e.g., LMDC, Port Authority, Mayor's Office) to participate and provide support (e.g., data, staff support, meeting spaces, funding) to the overall Scorecard effort. Also invite universities, such as those already participating in the Civic Alliance, to encourage faculty and students from appropriate schools or programs (e.g., public administration, planning) to provide support—or shift some of their existing support—to the scorecard effort.
Monitor the progress and how well resource needs are being met for each issue group, and provide facilitation or technical assistance from the Core Team as possible.
Raise funds from foundations or other sources, as needed, to complement in-kind efforts and sustain the Scorecard over time.

A Suggested Process for Developing an Issue Scorecard

Develop a 'basic framework' for the major Issue Scorecard, focusing on major initiatives and desired outcomes.
Provide an opportunity for public involvement to 'test' the framework and desired outcomes, develop them further, make adjustments as needed, and provide input for developing a weighting scheme for different elements of the framework. (A public engagement process might be organized collectively for all Issue Scorecards and the Overall Scorecard.)
Work with cognizant public agencies, as they are willing and able to cooperate, to develop timelines, milestones, and how to 'score' each initiative at different points on time. If key agencies will not provide full cooperation in developing timelines and milestones, civic groups with the knowledge of agencies' planning, budgeting, and capital and operating implementation cycles will project 'expected timelines' for each initiative.
Formulate explicit performance measures for all desired outcomes, especially to match outcomes articulated, refined, or confirmed in public engagement processes.
Find data sources for all performance measures. If data sources are not available for the exact measures desired, research potential sources to see if 'close proxy' measures are available, and use those. If no 'close proxy' measures are available, consider developing and using new data collection techniques for important measures, especially for measures or outcomes considered especially important to citizens.
Look for opportunities to involve citizens in performance measurement efforts (e.g., focus groups to help develop measures, citizen surveys for perception-related data, use of volunteer 'trained observers,' perhaps using handheld computers to record field observations, as in the Fund for the City of New York's COMnet project).
Obtain or collect baseline data for performance measures.
Set targets for future improvement for each measure based on when various initiatives will reach key stages of implementation, and contribute projected amounts to desired outcomes.

PROCESS FOR 'SCORING' RESULTS FOR EACH REPORTING PERIOD

Data Collection: Obtain the planning, funding, implementation, and performance data for the current reporting period for each initiative and performance measure on each issue scorecard.
Scoring Each Initiative: To the extent that objective data can be used to determine a definitive 'score' for each part of each initiative, it should be, to minimize possible questions on how a score was determined. For example, if $9 million has been approved in a public budget for a project that should have had $10 million approved by the end of the reporting period, then 90% of the 'funding points' would be scored for that initiative. In some cases, judgment calls may be needed to determine parts of a score. For example, an initiative may not quite have reached a key implementation milestone in a reporting period, but it came close. A judgment call could be made to give the initiative 'partial credit' for that milestone, based on an assessment by the oversight group of how close the milestone is to accomplishment, and reasonable progress is continuing to be made on the initiative.
Roll Up Initiative Scores and Issue Scores: When each initiative and performance measure for a major issue is scored for the reporting period, 'roll up' those scores into the total 'issue score' for that issue scorecard. Then, roll up all issue scorecard results into the overall 'Total Rebuilding Score' for the reporting period.
Possible 'Recalibration' of Scoring Systems: Before reviewing most data for a reporting period, each 'cognizant oversight group' will review its 'scoring system' to determine if it still makes sense, or developments since the last report suggest that the scoring system be 'recalibrated.' For example, a project's expected cost may go up significantly. The oversight group may need to adjust the funding targets over time for that project. A more complex case may involve, for example, a desired initiative built into the scoring system that has died in the state legislature, while a new initiative has emerged for the same purpose which has a different approval, funding, and implementation process. The oversight group will have to decide whether to score 'zero' for the original initiative, or to replace it with the new initiative, and provide a new scoring approach to the new initiative based on its specific approval, funding, and implementation process. If the recalibration being considered is 'technical' in nature (e.g., to account for changed cost estimates or changes in plans for an initiative), the oversight group would make the changes. However, some recalibrations would suggest changes in the relative importance of major issues or initiatives, such as a significant re-weighting of most initiatives in an Issue Scorecard, or the re-weighting of the major issues in the Total Scorecard. These re-weightings might be justified by changing circumstances over time. However, this kind of 'major recalibration' of scorecards should probably only be done infrequently (e.g., every three or four years) and only after some public engagement process is used to ensure the new weightings reflect citizen concerns.
Clear Disclosure of All Recalibration: Whenever a Scorecard has been recalibrated in any way, that fact must be clearly disclosed in all relevant versions of the next several Scorecard reports. Where practical, it can be useful to report scores using both the old and recalibrated scoring systems, for one or two 'overlap periods.' In the more detailed versions of Scorecard reports, explanations should be provided on how scorecards were recalibrated, with information on how to find full technical details. Web versions of Scorecard reports should include links to full technical details of recalibration.