Charity Navigator recently posted its 12th annual national study analyzing regional differences in the financial, accountability and transparency practices of the largest U.S. charities on its website.

The methodology involves compiling data from the close to 5,000 charities (about 62% of the total evaluated by Charity Navigator). The charities were assigned to 30 metropolitan areas across the U.S., and then the median value of a number of financial variables and the percentage of compliance for a number of accountability and transparency metrics are added together and ranked, within each area, as compared to the national median scores.  The point, according to Charity Navigator, is to “reveal possible differences in financial, accountability and transparency practices of the various philanthropic metropolitan markets.”

As one of the rating criteria is “CEO Compensation,” the first thought is that this might be an interesting source of comparative data on charity salaries. Here is a conclusion actually stated on the website:

“The fact that CEO Compensation tends to be greater in D.C. than Orlando is an important realization for the charities and donors in each of those cities. It doesn’t mean that D.C.’s charities are wasteful or poorly managed. Rather, it means that the cost of doing business in D.C. is greater. Recognizing these differences is essential to the effective management and evaluation of nonprofits.

There are many subjective decisions built into the dimensions and metrics used in this study, but, for this blog, let’s just focus on the usefulness of the compensation data presented. The table below shows the CEO compensation values for the two areas mentioned in the quote from the study above.

The medians are calculated using widely varying numbers of charities, characterized only as “large.”  It is widely recognized that one of the characteristics most influential on salary levels is size of organization — the larger the organization, the higher the salary. So without more information on the sizes of the charities used to calculate the medians within the geographic areas and with the number of charities included ranging from 684 in D.C. to 53 in Orlando, there could be many reasons for variation in the medians for these regions.

Although there is general information provided on the type of charities in each metro area, a breakout by type is not available for the median calculation.  This could also have a major impact on the resulting median.  For example, hospitals and universities tend to be among the largest and highest paying organizations; so, if there are many more in D.C. than there are in Orlando, the medians will not yield an “apples to apples” comparison – rather a “fruit salad” approach.  Without some kind of statistical method to adjust for different sizes and types of organizations, one should be wary of drawing any conclusions about the reasonableness of CEO compensation in a given area.

Finally, there is no evidence presented to support the conclusion that the difference in CEO compensation is because “the cost of doing business in D.C is greater.”  The criteria listed do not appear to include any measures of the cost of doing business – so this statement seems purely speculative.

So let’s review what the IRS requires when charity CEO compensation is determined – that is, comparisons of salaries in similar organizations (similar in size, similar in type, and maybe similar in geography).  ERI’s Nonprofit Comparables Assessor can be used to easily find that group of similar organizations, using the size and type criteria – and the resulting comparable data will be accepted by the IRS.

The table below shows the average compensation for a specific type of charity, Human Service Organizations, of different sizes (based on annual revenues), in the two locations, compared with the U.S. national average.  This is the type of detailed data that is needed to determine reasonable compensation.

Although Charity Navigator offers its 2016 Metro Market Study as “another valuable resource to help you make informed giving choices and learn more about the charities in your community,” let the user beware!  Check out ERI’s Nonprofit Comparables Assessor.