Tag Archives for benchmarking

Adding 401(k) Plans Whose Plan Year Begins in Q2, Q3, or Q4

The OnlyBoth/Benchmine engines will now include 401(k) plans whose plan year begins in Q2, Q3, and Q4. We will start with the plans in the $1B+ assets bracket, and update all the seven assets brackets in the coming days.

When first applying to 401(k) plans the OnlyBoth AI-powered  technology for comparative performance analytics & reporting, we opted to limit the scope of the BenchMine tools to plans whose plan year began in Q1 (Jan-Mar), which covered about 95% of them. Our reasoning was that a one-year comparison of plans starting in Q1 to plans starting, say, in Q3 would not be a fair comparison, because highly external factors such as the stock market could lead to, say, an inferior performance during that calendar year but a superior performance during the next (or previous) year. Enlarging the comparison to multiple years could make it less unfair but would obscure the yearly plan performance.

This author appeared on a panel on Benchmarking Investments and Fees at the 2025 PLANSPONSOR National Conference in Chicago. In the audience was a plan advisor who, after the panel ended, came to offer compliments on the free and open tools available at BenchMine.com. During a subsequent conversation, the advisor asked about a plan that was not in the BenchMine data, and I checked and answered that its plan year began in Q2. After further discussion, I began to wonder: “Can such plans be included while making adjustments to minimize the unfair comparisons?”. After some deliberation, the answer turned out to be “Yes”.

Here are the key steps to the inclusion approach. 

  1. First, add the symbolic attribute “plan year began in”, with possible values Q1, Q2, Q3, or Q4 to every plan.  This will enable doing simple searches at the Discovery Engine such as “List the Q2 plans by highest total administrative expenses.”
  2. However, prevent this new attribute from being used within benchmarking insights, e.g., “This plan had the highest total administrative expenses of all the Q2 plans in the Manufacturing sector.” which seems less insightful or interesting than other peer comparisons.
  3. Identify those plan measures that are most sensitive to external factors. We selected the measures that involve earnings, income, and yield.
  4. For Q2, Q3, and Q4 plans, replace those sensitive measure values with N/A, to avoid comparisons to other plans with a different Q (quarter).  However, let users easily see the values in the plan’s profile, which will state something like “net income ($X, but changed to n/a because the value of ‘plan year began in’ must equal ‘Q1’).”

Note that for plan years that begin in Q2, Q3, or Q4, performance measures relating to earnings, income, and yield will not be used for comparison in the Benchmarking Engine. This means that the Scoring Engine results for such plans will be less comprehensive, since the benchmarking scores for such plans will rely on administrative and operational aspects such as expenses, contributions, asset diversity, balances, liabilities, corrective distributions, insurance coverage, and possible fraud.

We believe that the above approach will bring added value to the benchmarking of 401(k) plans, while minimizing the introduction of unfair or misleading comparisons.

Raul Valdes-Perez

New Benchmine Measure to Compare Investment Returns of Employer 401(k) Plans

Benchmine.com, provided by OnlyBoth, launched in late November a free, open-to-all website for comparing the performance of 55,000+ employer 401(k) plans, offering many novel analytic capabilities to users.

The federal data source (Department of Labor EBSA) reports many core measures of 401(k) plans, leaving it to users of the data to introduce derived measures that better enable broad performance comparison across plans of different sizes. This led to our incorporating total administrative expense ratio, defined as total administrative expenses (Line 2i(5)) divided by total assets (Line 1f), times 100. Note: all data sources referenced here are from Schedule H.

The Benchmine 401(k) engines needed a measure of plan-year investment returns that likewise enables fair comparison. The federal data reports on total income and net income during a plan year, but these include employer and participant contributions and rollovers, which tend to skew the returns on investment. Also, plans sometimes transfer investment assets out of, or into, the plan, which also complicates fair comparison.

After consulting 401(k) industry experts, we decided on a new measure yield on beginning-of-plan-year total assets (yield for short), defined as net earnings on investments (the sum of the 10 column (b) entries from section 2b minus investment advisory and management fees from Line 2i(3)) divided by total assets at the beginning of the plan year (Line 1f(a)), times 100. By itself, this doesn’t deal with the complications of mid-year asset transfers (section 2l), so we added a qualifying criterion that a plan’s asset transfers (incoming + outgoing) be less than 1% of its total assets at the beginning of the plan year. This prerequisite disqualifies about 5% of the 55,788 401(k) plans at Benchmine, which then get a value of N/A for their yield.

These three CY 2021 examples of employer 401(k) plans (names omitted here), from different total-assets brackets, stand out on their joint yield and administrative expenses:

  • Only PLAN (within the $10M-$50M bracket) has both such a high total administrative expense ratio (1.716%) and such a low yield (11.70%).
  • In California with its 209 ($250M-$1B) plans, only PLAN has both such a high yield (18.31%) and such a low total administrative expense ratio (0.002%).
  • PLAN has the highest total administrative expense ratio (0.360%) among the 196 ($100M-$250M) plans that have 1,000 to 4,999 total participants and have at least a 16.66% yield.

In conclusion, Benchmine is now equipped with good measures for both administrative expenses and investment returns, all in the service of enabling fair comparison, heightening performance transparency, helping to drive improvement, and empowering participant choices.

Raul Valdes-Perez

Benchmarking the Tax Systems of 195 Countries

Taxation at the national level is controversial. Economists gather data and form opinions, and so do politicians. Factual, comparative insights on worldwide tax systems are needed. So we applied an automated Benchmarking Engine (taxes.onlyboth.com) to tax data on 195 countries, uncovering 7,617 insights or about 39 insights each, in perfect English, all automated.

paying taxes

Paying the Tax (Collector), by Pieter Brueghel the Younger

The U.S. Agency for International Development publishes a fascinating Collecting Taxes Database on the tax systems of the world’s countries.  The database expresses 33 attributes relating to various metrics and traits, relating to tax rates, efficiency in collecting the revenue that the rates target, diversity in sources of tax revenue (VAT, personal income, corporate, etc.), tax administration, and so on.

We downloaded the latest available version (2012-2013) as well as an earlier 2009-2010 version, in order also to express changes over a three-year interval and enable benchmarking on trends.

The U.S. corporate-tax system is controversial because of its very high rate.  Does the engine find any noteworthy insights relating to corporate taxation?  Indeed it does, which we’ll quote at length:

USA has the lowest corporate income tax productivity (0.07) of the 32 nations with at least 9.3% personal income tax collection as a percentage of GDP (USA is at 11.8%). That 0.07 compares to an average of 0.24 and standard deviation of 0.24 across the 32 nations.

Reaching the average of 0.24 would imply a total increase of 5.9% (absolute) in corporate income tax collection as a percentage of GDP.

USA has these standings among those 32 nations:
corporate income tax collection as a percentage of GDP = 2.6% (8th-least)
corporate income tax rate = 35.0% (most overall)

trailed France (0.08), Malawi (0.08), Austria (0.09), and Belgium (0.09), and others, ending with Algeria (0.86).

1 out of the other 31 nations was ruled out due to missing, unknown, or not-applicable values for corporate income tax productivity, i.e., Angola.

Let’s interpret this. First, it says that the U.S. has the highest corporate income tax (35%) in the world. However, this high rate leads to a low revenue outcome, as indicated by the low 0.07 productivity score.  The Collecting Taxes Database calculates this corporate-income-tax productivity by “dividing the ratio of total corporate income tax revenues to GDP by the general corporate income tax rate.”

Not only is productivity low, but it’s the lowest of the 32 nations that collect a significant share (at least 9.3%) of personal income in relation to GDP (gross domestic product). It’s lower than France, Malawi, and others.  Here’s a plot:

U.S. corporate income tax productivityA separate insight reveals that the U.S. has the lowest corporate-income-tax productivity of the 17 nations with an agriculture sector as a percentage of GDP of at most 1.6% (the U.S. is at 1.2%).

Now let’s click on What’s best in class? to see what the U.S. could aspire to, as shown by nations that are similar, i.e., whose overall values in the database are most similar to the U.S.  It turns out that Hong Kong and South Korea do best among the 20 countries most like the U.S.

Hong Kong has the highest corporate income tax productivity (0.32) among the 20 nations most similar to USA (with 0.07) that likewise have a high-income economy.

Next with 0.18 is South Korea.

USA is 35th best among the 40 nations with applicable values and that have a high-income economy, which range from a worst of 0.02 (Bahrain) to a best of 0.99 (Qatar), with an average of 0.20 and standard deviation of 0.21.

Among all 164 nations with applicable values, the overall average is 0.15 and standard deviation is 0.16. Best is Qatar, with 0.99.

As is typical of a benchmarking engine, we can leave it to human experts – economists and political leaders in this case – to figure out whether the U.S. has a tax problem, what’s causing it, what are possible solutions, and which solution is best.  Our aim has been to provide this Taxes Benchmarking Engine as a public service and as a showcase of what automated benchmarking can do, as was done earlier for college financials, hospitals, and nursing homes.

Benchmarking need not be taxing. Simply enter any country at taxes.onlyboth.com and see how it’s doing, where it could improve, what’s trending, and what’s best in class.

Raul Valdes-Perez

 

 

Relaunch of Hospitals Benchmarking Engine

OnlyBoth benchmarks U.S. hospitals as both a public service and as a visible demonstration of the power of an automated Benchmarking Engine. This enables hospital stakeholders to instantly discover in perfect English how they’re doing, not compared to absolute standards or arbitrary peers, but to all peers and groups.

We launched our first version this summer. Today we relaunched our hospitals benchmarking engine based on fresh data and technical advances:

  1. updated Hospital Compare dataset from Medicare.gov, now on 4,803 hospitals
  2. new hospital attributes relating to hospital performance and geography
  3. better expression of key types of insights
  4. improved heuristics leading to more insights per hospital
  5. addition of data on hospital networks, enabling intra-network comparisons

1.  We have refreshed the data in the hospital application based on a late-September data release at the Hospital Compare data download page. This new release also contains new hospital attributes, as discussed below.

2.  Since geography is an important determinant of peer groups, we’ve added attributes that enabling grouping East Coast, Southern, and Western states. We’ve also added two new attributes from the updated Hospital Compare data that relate to deaths or unplanned readmission due to coronary artery bypass grafting (CABG) surgery, and five new attributes that express hospital-readmission ratios for various afflictions.

3.  A key type of insight expresses how entities that are within an elite peer group fall short along some key dimension. For example, our recent Harvard Business Review article, which explains why benchmarking is done wrong and how to do it right, gives this example of Stanford Hospital:

None of the other 344 hospitals with as many patients who reported YES, they would definitely recommend the hospital (85%) as Stanford Hospital in Stanford, CA also has as few patients who reported that the area around their room was always quiet at night (41%). That is, among those 344 hospitals, it has the fewest patients who reported that the area around their room was always quiet at night.

As the saying goes, this was too clever by half. After considering feedback from surveying users, this insight now appears, with the refreshed data, like this:

Stanford Hospital in Stanford, CA has the fewest patients who reported that the area around their room was always quiet at night (40%) among the 811 hospitals with at least 80% of patients who reported YES, they would definitely recommend the hospital (Stanford Hospital is at 84%). That 40% compares to an average of 69.4% and standard deviation of 10.7% across the 811 hospitals.

Of course, this improvement affects thousands of insights, and millions in the future.

4.  We’ve improved the heuristics that enable finding valuable needles within the huge haystack that results from taking multiple slices out of a dataset of half a million hospital attribute values. Our new Hospitals Benchmarking contains 522,142 insights, or around 109 insights per hospital, compared to the previous 101 per hospital. The key benchmarking question – Where can this hospital improve? – has seen a 4% increase in answers per hospital.

5.  For a hospital-network executive, it’s valuable to benchmark individual hospitals against others in the network, especially because knowledge transfer of good practices can happen more easily when two entities have the same owner. We’ve added a parent attribute that for now includes four networks:  UPMC, Kaiser Foundation, Texas Health Resources, and NYC Health and Hospitals. We’ll add other hospital networks over time.

We expect that this hospitals application, and the diffusion of benchmarking engines in general, will further the goal of enabling universal betterment through data-driven comparison with peers, greatly simplified in terms of human work, but greatly expanded in terms of action-provoking insights.

Raul Valdes-Perez

Benchmarking Financials

Today we have launched a second public showcase application of our Benchmarking Engine, this time to mostly Department of Education IPEDS data on U.S. post-secondary educational institutions, or private colleges for short, that follow the FASB accounting standard, ranging from Harvard to the Belle Academy of Cosmetology. The financials data is from FY 2013, the latest available from IPEDS.

The 1,889 private colleges are described with 151 mostly-financial attributes, of which 101 are dollar amounts (investment, spending, debt, liability, etc. and their sub-categories) and 11 are financial ratios, augmented by some college rankings and profile and type attributes.

Given its emphasis on internal financial metrics, this benchmarking application addresses the core benchmarking questions from an institutional viewpoint, not from a student or faculty point of view. Some value judgments were made, for example that less debt is better than more debt, but of course in some circumstances more debt can be good, such as when the interest is low and the return on the debt is high.

Here is an example insight on how Columbia University can improve:

None of the other 1,631 private colleges with as few total liabilities ($3.028B) as Columbia also has as much debt related to property, plant, and equipment ($1.479B).  That is, it has the most debt related to property, plant, and equipment among those 1,631 private colleges.

Columbia-liabilities-debt-OnlyBoth`

On a related note closer to home, here’s a rather favorable insight about Carnegie Mellon:

In the Mid Atlantic with its 434 private colleges, only Carnegie Mellon both spends as much on research ($284.3M) and has as few research expenses – operation and maintenance of plant ($9.958M).

Clearly, software, psychology, and decision science don’t cost much! Over on the west coast, Stanford is seen to have rich, forthcoming donors:

Stanford has the most private gifts ($694.5M) among all 1,889 private colleges. Those $694.5M represent 4.3% of the total among all 1,889 private colleges, whose average is $8.632M.

As a final example, let’s move southwest and to smaller colleges, for example Austin College in Texas:

In the Southwest with its 101 private colleges, only Austin College has both as much construction in progress ($32.95M) and as few total assets ($251.5M).

Build it and they will come!

Accounting statements and financials in general are an especially promising application of Benchmarking Engines, because financial metrics follow established standards – FASB in this case – and relate to critical organizational performance.

Enter your own private college here.

Raul Valdes-Perez

Ranking SaaS Vendors by their Benchmarking Activity

As I’ve argued elsewhere, business benchmarking has been held back by the problem of data availability, as well as by the lack of software automation, despite its worthy goal of enabling continuous organizational improvement.

benchmarking-saas

Most SaaS vendors are uniquely placed to sidestep the availability problem, because SaaS generates rich data as a byproduct of serving its customers. This data can be captured by vendors and put to good use, for the benefit of those same customers, via benchmarking. The exceptions tend to be utility-like SaaS, whose customers only care whether the service is on or off, or vendors who have little visibility into how customers perform the business process that their services support.

So how well are SaaS vendors exploiting this emerging opportunity?  To find out, we analyzed the benchmarking activity of the Montclare SaaS 250 – the 250 “most successful SaaS companies in the world” according to Montclare, self-described as the “Industry’s Premier Research and Consulting Firm Focused on SaaS.”  For each vendor, we measured benchmarking focus by dividing the number of its website’s hits on the query benchmarking by its total number of webpages, both as reported by the Google API.  Below are the ranked results, which range from 0% to 94%.  We opted to leave untouched a few anomalous results due to hits from hosted content, e.g., at Google and at LinkedIn.

Overall, there’s lots of activity. SaaS is a busy playing field for benchmarking. Unanswered here is whether that activity reflects actual vendor benchmarking services or something else. Also not addressed is whether vendor benchmarking is powered by automation.

Interestingly, SaaS pioneer Salesforce.com comes in below at #221. Benchmarking on its website tends toward blog topics or partner activity, not Salesforce’s own offerings.

Raul Valdes-Perez

  1. Veracode 94.08%
  2. Tangoe 87.13%
  3. IQNavigator 80.62%
  4. Meltwater Group 70.14%
  5. athenahealth 64.66%
  6. SciQuest 53.85%
  7. MediData Solutions 53.48%
  8. ON24 26.37%
  9. ComScore 25.95%
  10. Intel 25.69%
  11. Apptio 25.3%
  12. ServiceSource 20.42%
  13. Symantec 16.41%
  14. Deltek 15.14%
  15. GTNexus 14.95%
  16. Xactly 14.01%
  17. Blackbaud 13.9%
  18. Jobvite 13.21%
  19. Trend Micro 12.81%
  20. Synygy 12.74%
  21. Coupa Software 12.52%
  22. AlphaBricks 12.5%
  23. Domo 11.85%
  24. ADP 11.54%
  25. Beckon 10.78%
  26. Marin Software 10.45%
  27. Intacct 10.0%
  28. Act-On Software 9.51%
  29. Peoplefluent 9.03%
  30. E2open 8.62%
  31. CallidusCloud 7.86%
  32. Amber Road 7.8%
  33. Fleetmatics 7.64%
  34. Demandware 7.3%
  35. Instart Logic 7.22%
  36. Reval 7.22%
  37. Wolters Kluwer 7.01%
  38. Globoforce 6.83%
  39. 3D Systems 6.82%
  40. Marketo 6.72%
  41. eGain 6.5%
  42. RingLead 6.5%
  43. Achievers 6.14%
  44. FICO 6.13%
  45. CRMnext 5.82%
  46. Veeva Systems 5.79%
  47. KnowledgeTree 5.73%
  48. Basware 5.61%
  49. Deem 5.52%
  50. Cornerstone OnDemand 5.4%
  51. Bullhorn 5.34%
  52. LiveOps 5.19%
  53. Tidemark 5.03%
  54. Hubspot 4.91%
  55. Lattice Engines 4.9%
  56. MindTree 4.87%
  57. Telogis 4.87%
  58. Plex 4.69%
  59. InsideView 4.48%
  60. Cloudpay 4.46%
  61. Monitise 4.44%
  62. Nice Systems 4.27%
  63. Birst 4.25%
  64. Payscale 4.24%
  65. inContact 4.23%
  66. NewVoiceMedia 4.19%
  67. Anaplan 4.16%
  68. PROS Holdings 4.08%
  69. Zuora 4.01%
  70. New Relic 3.99%
  71. Mimecast 3.97%
  72. Qualys 3.88%
  73. GoodData 3.86%
  74. FinancialForce.com 3.8%
  75. Insidesales.com 3.75%
  76. Actian 3.73%
  77. Cerner Corporation 3.66%
  78. CSC 3.66%
  79. Healthstream 3.66%
  80. MYOB 3.64%
  81. Adaptive Insights 3.6%
  82. Gainsight 3.6%
  83. ClearSlide 3.55%
  84. Verint Systems 3.52%
  85. Oracle 3.45%
  86. Lumesse 3.38%
  87. Ultimate Software 3.33%
  88. AppDynamics 3.26%
  89. Kronos 3.24%
  90. Ramco Systems 3.2%
  91. Halogen Software 3.18%
  92. RightScale 3.13%
  93. Descartes Systems 3.12%
  94. Workday 3.09%
  95. Fujitsu 2.98%
  96. NetSuite 2.93%
  97. Ceridian 2.89%
  98. QuestBack 2.88%
  99. Ericsson 2.84%
  100. Dassault Systèmes 2.8%
  101. Rocket Fuel 2.79%
  102. Nuance Communications 2.7%
  103. DealerTrack 2.66%
  104. Selectica 2.6%
  105. Survey Monkey 2.57%
  106. AdRoll 2.54%
  107. Opower 2.52%
  108. Saba 2.52%
  109. iCIMS 2.5%
  110. Intuit 2.48%
  111. Rally Software 2.44%
  112. Blackline Systems 2.38%
  113. Host Analytics 2.37%
  114. eVariant 2.36%
  115. Covisint 2.34%
  116. Apttus 2.32%
  117. Proofpoint 2.3%
  118. VMware 2.3%
  119. cVent 2.25%
  120. EMC Corporation 2.24%
  121. Epicor 2.24%
  122. ServiceMax 2.23%
  123. CashStar 2.09%
  124. SAS Institute 2.08%
  125. SugarCRM 2.08%
  126. Infor 2.03%
  127. OpenText 2.0%
  128. SPS Commerce 1.95%
  129. WebTrends 1.94%
  130. Akamai Technologies 1.93%
  131. DATEV eG 1.89%
  132. FPX 1.82%
  133. Hitachi 1.81%
  134. Huddle 1.81%
  135. Threatmetrix 1.8%
  136. BroadVision 1.79%
  137. Kyriba 1.79%
  138. Support.com 1.71%
  139. Castlight Health 1.68%
  140. Atlassian 1.65%
  141. Workforce Software 1.65%
  142. Bottomline Technologies 1.6%
  143. Brightcove 1.6%
  144. Retail Solutions 1.57%
  145. 2U 1.51%
  146. Five9 1.5%
  147. LinkedIn 1.41%
  148. Hyland Software 1.4%
  149. Workfront 1.39%
  150. Informatica 1.34%
  151. Mulesoft 1.33%
  152. SilkRoad 1.31%
  153. IBM 1.22%
  154. Mix Telematics 1.22%
  155. BenefitFocus 1.18%
  156. Blue Jeans Network 1.18%
  157. MicroStrategy 1.11%
  158. Trustwave 1.1%
  159. Google 1.09%
  160. TIBCO Software 1.09%
  161. Xero 1.09%
  162. Blackboard 1.08%
  163. Silver Spring Networks 1.08%
  164. Zendesk 1.04%
  165. AeroHive Networks 1.01%
  166. Alfresco 1.01%
  167. Clarizen 1.01%
  168. GitHub 0.98%
  169. Jive Software 0.98%
  170. Paychex 0.98%
  171. ASG Software 0.97%
  172. Cision 0.96%
  173. Freshbooks 0.95%
  174. Logik 0.94%
  175. Practice Fusion 0.94%
  176. Autodesk 0.92%
  177. SolarWinds 0.89%
  178. Pegasystems 0.88%
  179. Digital River 0.87%
  180. Siemens 0.86%
  181. Constant Contact 0.84%
  182. LivePerson 0.84%
  183. Synchronoss 0.81%
  184. Dell 0.78%
  185. Citrix 0.77%
  186. Opera Software 0.76%
  187. Hewlett-Packard 0.75%
  188. Tableau Software 0.75%
  189. Avangate 0.66%
  190. Paylocity 0.65%
  191. Mindjet 0.64%
  192. Cisco Systems 0.63%
  193. Aria Systems 0.62%
  194. Hightail 0.62%
  195. Glassdoor 0.6%
  196. Nakisa 0.6%
  197. Okta 0.6%
  198. Deluxe Corp 0.57%
  199. ChannelAdvisor 0.56%
  200. FrontRange 0.54%
  201. CA Technologies 0.53%
  202. Daptiv 0.51%
  203. SAP 0.51%
  204. ServiceNow 0.51%
  205. BMC Software 0.5%
  206. IntraLinks 0.5%
  207. Splunk 0.49%
  208. Finnet Limited 0.47%
  209. Bill.com 0.46%
  210. Limelight Networks 0.46%
  211. Box 0.44%
  212. Zoho 0.42%
  213. Adobe Systems 0.41%
  214. CollabNet 0.41%
  215. SugarSync 0.41%
  216. MobileIron 0.39%
  217. Lithium Technologies 0.32%
  218. RingCentral 0.32%
  219. Twilio 0.32%
  220. Elance/oDesk 0.3%
  221. Salesforce.com 0.29%
  222. Zscaler 0.28%
  223. Magic Software Enterprises 0.27%
  224. Microsoft 0.25%
  225. Jitterbit 0.23%
  226. Parallels 0.23%
  227. Bazaarvoice 0.17%
  228. Basecamp 0.16%
  229. Active Network 0.15%
  230. M-Files 0.15%
  231. DocuSign 0.14%
  232. LogMeIn 0.14%
  233. DropBox 0.11%
  234. Rocket Lawyer 0.11%
  235. Doximity 0.08%
  236. Ping Identity 0.06%
  237. BorderFree 0.04%
  238. Evernote 0.04%
  239. TOTVS 0.04%
  240. Exact Holding NV 0.03%
  241. Arena Solutions 0.0%
  242. Carbonite 0.0%
  243. Cybozu 0.0%
  244. Eventbrite 0.0%
  245. j2 Global 0.0%
  246. KDS 0.0%
  247. META4 0.0%
  248. Paycom 0.0%
  249. Xtenza Solutions 0.0%
  250. Vend 0.0%