A Study in Scientific Method: Consumer Reports’ Orbit Carseat Testing

FacebooktwittermailFacebooktwittermail

I subscribe to Consumer Reports.  I enjoy the magazine.  Every once in a while, it does seem that a recommendation for or against a particular product comes out of thin air.  That’s true of all media reviews, CarseatBlog being no exception, of course.  Unfortunately, it seems that their carseat reviews have had more than their share of mistakes and questionable advice.  The biggest gaffe was in 2007, when they failed to catch a mistake and published the results about carseats that failed a side impact test.  Heather has an article about this.  By the time they admitted their error, a lot of damage had already been done.  Google still indexes numerous headlines and stories from the original flawed test and consumers may well find those articles without learning about the retraction.  Maybe larger companies can absorb the backlash from a rating of “Don’t Buy: Safety Risk”.  For smaller companies, something like that can do a lot of damage to the company and its employees.  For this reason, scientific method really must be closely followed and any results that could indicate a risk to consumers (and damage to a company’s reputation) must be checked, rechecked and triple checked for any possible error.

In August of 2009, Consumer Reports struck again.  This time, it was a single model, the Orbit Baby Infant seat.  Kecia has a great summary of this report.  Consumer Reports may be right this time.  Or they may be wrong.  They simply haven’t developed a trustworthy reputation in the child passenger safety area, so it’s difficult to give them the benefit of the doubt.  Their test results can create a lot of hype, a lot of panic and a lot of publicity for their magazine and website.  The disturbing part is that their press release seems to eschew scientific method, leaving too many variables and conditions ignored.  After their flawed test and subsequent apology in 2007, I would have expected every “i” dotted and every “t” crossed before they released another test result.  From what I have read on their website, they have left too many questions for me to believe their results without further testing.  Specifically:

1) After the failures, they tested the seat 4 more times without a failure.  No results for other models were issued.  Were other models tested 6 or more times?  Did any fail?  It’s hard to determine if there is a problem with the product or the testing without that information.

2) They admitted that the manufacturers instructions were not followed in the failed crash tests.  The Orbit Infant Seat is a unique product and has unique installation requirements.  Were any tests done with the seats installed according to the instruction manual?  What were the results?

3) Could the laboratory or test parameters be at fault?  They didn’t name the lab or personnel who did the testing.  Were the results confirmed by an independent laboratory or agency?

4) Could a defect be involved that affected only some product?  Were the units that failed from a different date range or batch?  Have they been examined for defects?

5) Given the issues with the 2007 crash testing, have the details of the testing and the complete results been published and peer reviewed by independent experts?

These types of questions should be considered as part of any testing that follows established principles of scientific method.  I certainly hope they were asked before this warning was published.  Certainly, they should have been both asked and answered, given what happened in 2007 and the subsequent apology by their president, Jim Guest, where he said, “I apologize on behalf of Consumers Union and I promise you we’re working hard to ensure that such an error does not happen again.”

It may turn out that Consumer Reports is right.  Hopefully, an independent review of this issue will prove or disprove their findings.  In the mean time, Orbit Baby has already conducted such tests, using the same laboratory and conditions and released a new statement:

Dear Orbit Baby parent, 

As I mentioned in my last letter to you in regards to the Consumer Reports testing of the Orbit Baby Infant Car Seat,  we referred this matter to NHTSA, the government agency responsible for car seat safety.  I am pleased to share the following information:

Update September 1, 2009:  The National Highway Traffic Safety Administration (NHTSA) tested the Orbit Baby Infant Car Seat for compliance with all crash testing requirements of Federal Motor Vehicle Safety Standard (FMVSS) 213NHTSA confirmed that the Infant Car Seat passes these tests.NHTSA also independently verified:

  • The middle shoulder harness slot is the appropriate slot for the size of the official CRABI 12 test dummy.
  • The knob on the car seat Base should be turned to engage the StrongArm™ tightening mechanism to achieve the regulated seat belt tension for compliance testing.

Therefore, our Infant Car Seat not only passes our own tests in independent labs, it also passes tests conducted by NHTSA, the government agency that regulates traffic safety.

Orbit Baby maintains that Consumer Reports did not properly follow the Infant Car Seat instructions when they conducted their tests.  (Our full response can be found here.)  As an Orbit Baby parent, I believe that this issue highlights the importance of following car seat instructions, as well as the need for parents to educate grandparents, babysitters, and other caregivers on proper car seat usage.

Please be sure to take advantage of these helpful resources on our website:

Thank you for your support of Orbit Baby.

Sincerely,

Joseph Hei

CEO and Co-founder of Orbit Baby

 

Consumer Reports has the resources to do a great service to parents in regard to additional testing of child restraints.  It’s too bad this test seems seems akin to “Crying wolf.”  Even if they are right, a third party verification is now needed to prove it.  If it is eventually shown that they have made another mistake, I sure hope their board of directors takes a very close look at how their testing is done and the serious impact their press releases can have on businesses and people.

As for our part, we plan to have a full review of the Orbit Baby Infant and Toddler system later this month.  We won’t be able to tell you about crash test results, but we will be able to tell you a lot about this innovative child restraint!

4 Comments

  1. CPSDarren September 4, 2009
  2. Jenny September 4, 2009
  3. Bookmama September 3, 2009
  4. kate September 3, 2009