Types of Data—Quantitative and Qualitative
To provide a picture, from an asset-based perspective, of both policy and performance at the state level, CFED gathered two different kinds of data for the Report Card. For the Asset Outcomes Index, we used primarily quantitative data that are relatively easy to rank, present, and understand. For the Asset Policy Index, primarily qualitative data were used. Because these qualitative data present particular challenges with respect to ranking states, presenting the data in a manner that is easily comprehensible, and comparing the data across both indexes, a few words should be said about the policy data in the Report Card.
First, the process of identifying the most effective public policies in a particular policy area, such as housing or small business development, is complicated. Determining conclusively which policies actually “work,” and for those that work, which ones truly benefit low-income populations, can be a huge academic undertaking. Second, even when it is clear which policies make a difference, problems in measurement arise:
- Even for the same policy, such as a Capital Access Program, it can be difficult to capture fully the design, scale, scope, and quality of implementation, all of which are critical.
- It can be difficult to decipher whether a state-level program, which is not necessarily operated by the state, is still a state responsibility and worth “counting.”
- Comparable 50-state data are not always available.
The particular policies selected for inclusion in the Assets Policy Index by CFED were identified by the Report Card’s advisory committee, conversations with experts, and CFED’s knowledge of asset-based policies that are the most promising, proven, or effective. In some cases, these policies were evaluated based on one or more thresholds or minimum standards to allow some measure of their effectiveness and enable ranking among states with that policy. In other cases, states earned credit for a particular policy on a “yes/no” basis, that is, either the state had a policy or it did not.
Finally, because of the different kinds of data used in the two main indices, one should be hesitant to directly compare the grades in the Asset Outcomes and Asset Policy Indices. First, many promising policies measured here are not yet being implemented to scale. Second, some of the current outcomes reflect the legacy of past policies rather than current ones. Finally, each index incorporates a broad range of indicators—from measures of homeownership to health insurance coverage—that are not all directly related.
