A social impact bond investment by Goldman Sachs in early childhood education that resulted in a six-figure payout for the Wall Street firm in its first year is raising questions about whether the program achieved the success it claimed, the New York Times reports.

With investments totaling $7 million from Goldman Sachs and the Chicago-based J.B. & M.K. Pritzker Family Foundation, the program, in partnership with the United Way of Salt Lake, financed the Utah High Quality Preschool Program, which is  aimed at reducing the number of at-risk children who require special education in K-12. In October, Goldman announced that of the one hundred and ten four-year-olds who attended preschools in the program during the 2013-14 school year that were identified as likely to need special education, only one required special education services in kindergarten. The successful outcome triggered the first investor payment for a social impact bond in the U.S. — a $260,000 payout that is expected to be the first of many, based on the number of students in the program who avoid special education each year through sixth grade. According to Goldman, Utah saved $281,550 in special education costs in the first year of the program, with investors in the program receiving the equivalent of 95 percent of those savings.

Early childhood experts told the Times, however, that even well-funded preschool programs — which the Utah program was not — typically have been found to reduce the number of students needing special education later by 10 percent or 20 percent, and rarely by more than 50 percent, suggesting that there may have been irregularities in how the Utah program's success was measured. For example, the program screened low-income three- and four-year-olds using a picture-and-vocabulary test known as the PPVT and labeled all those who scored below 70, a very low score, as being likely to require special education. According to nine early childhood education experts who reviewed the results for the Times, however, the PPVT isn't typically used to screen for special education, especially on its own, and there was little evidence for assuming that all children who scored poorly on the test — 30 percent to 40 percent of the children in the program, many of whom did not speak English at the time of testing — would require special education after preschool.

"To just assume that all these children would have gone to special education is kind of ridiculous," said Ellen S. Peisner-Feinberg, a senior scientist at the Frank Porter Graham Child Development Institute at the University of North Carolina at Chapel Hill. The lesson, added Peisner-Feinberg, is that before a pay-for-success program is considered again, investors and schools need to find better ways to measure their outcomes. "You have to be sure you have very rigorous ways of measuring the impact to make sure that it's legitimate in terms of the outcome you get. That didn't happen here."

A Goldman spokesperson told the Times that the program had adopted the methodology developed by the state and local school district. But Goldman, the Times notes, gave itself plenty of leeway for a return on its investment: as long as 50 percent of the children in the program avoided special education after preschool, it would earn back its investment, plus 5 percent interest.

Kenneth A. Dodge, a professor at the Sanford School of Public Policy at Duke University who has been an advocate for the pay-for-success model, told the Times that if the model is to succeed, it will have to be implemented differently than it was in Utah. "It is a step in the right direction," said Dodge, "but this is not the criteria I hope we hold ourselves to ultimately."