Mods Face Off Testing Methodology Q&A

[CoolVanilla]

Last updated April 13, 2008

Q: Generally speaking, how can the Mods Face Off Results sometimes be so different than results released by manufacturers?
The Mods Face Off events are not trying to prove or disprove manufacturers' claims with regard to the results of their products. Rather, our whole goal is to show what can be expected, on any given average day, when you stomp on the gas. Our testing methodology, in fact, directly reflects this goal:

• We standardized on 2 gear pulls as that is how our LX's naturally want to run. The industry standard is to make a single gear pull.
  • The Mopar TCM and CMR tuning has now allowed for single gear pulls. When possible, we try to use these in order to get those single gear pulls as that has proven to be what most folks want to see. This is taken on a car by car basis, but will always remain consistent for a given test.
• We'll fill up at the local gas pump where octane rating can legally vary plus/minus 3 from the advertised rating. The industry standard is to ensure a specific octane while testing.
• We use inertia mode on the dyno. The industry standard is to test in loaded mode.
• We make no attempt to control our climate; except to keep that environment as consistent as possible when deriving relative results on test day (our variables will not vary between tests). The industry standard is to regulate temperature, pressure and humidity for testing.
• We publish numbers averaged between three consecutive pulls as "official" Mods Face Off results. The industry standard is to publish the best numbers found for a given test.
• We use fans for cooling purposes only. The industry standard is to provide airflow equivalent to a car at speed.
• We focus on relative results. The industry standard is to focus on global results.

This point cannot be overstated: the Mods Face Off philosophy is simple. We attempt to answer the question "What can I expect from my daily driver when I stomp the gas?" Our tests are specifically (and uniquely) tailored to answer that exact question. Our tests simply will NOT yield what any given mod, under ideal conditions is capable of. The manufacturers tests will provide that information. Comparing the two is really akin to comparing apples to oranges. You owe it to yourself to consider both sets of results when deciding where to spend your modding dollar.

Q: What qualifications do you have to be dynoing products? Do you own a dyno shop? Are you in the automotive industry?
None, no and no! None of us had any qualifications what so ever! But, I think that's the beauty; we had no biases going into this.

We made our interests painfully clear 2 years ago, when we started all this. There were a bunch of people asking the question, "I appreciate what the manufacturer says, but what does this mod really do, for me at a stoplight?" I spoke up and said, "Gee I dunno... how hard can it be to find out?" <------MY FIRST MISTAKE!

From there, we became point persons for the experts, the OEMs and the like. We opened it up (and still do!) to all aspects of experience and qualifications, and then condensed the wealth of information and experience down into something that the 'common man', just like us, can understand and apply. We take the same approach the open source community does; its the community effort that has resulted in the test and resulting valuable info we have today. I hope it continues to be the community effort that will further shape, refine and therefore produce even more useful tests.

Q: What dyno do you use, and why?
The Mods Face Off dyno is a SuperFlow SF840 Chassis with full instrumentation, located at SVS R&D in Sacramento CA. This dyno is absolutely state of the art, and its two 400 horsepower eddy current drums are among the most sensitive available today.

This particular SuperFlow can operate in either "inertia" or "load bearing" modes; and within those modes we can choose various sub modes; "simple inertia", "controlled acceleration", "steady state control" and "road load".

A major benefit of the SuperFlow is its eddy current rollers; they allow us to actually measure both force and acceleration. Other systems we evaluated measure one or the other, and calculate the number it doesn't measure. They calculate this value based on manually input values for weight of the vehicle; and since weight can only be estimated at the time of test, this opens the door for an added variable (of course, we could specify the same weight always; but the car may not actually weigh that weight at the time; keep in mind the general rule of 100lbs = 1/10th at the track, and you can see that its critical to get this number right each and every time, or do as we do and just measure both values and not worry about the actual weight dyno day). Both force and acceleration are required for computing our power numbers, and by measuring both, the SuperFlow can deliver extremely accurate and consistent results.

We were also drawn to the experimentally proven accuracy of this dyno and to that effect we ensure the SuperFlow has been professionally calibrated just before each test event. Advertised accuracy is plus/minus 2%, however actual measured accuracy is sub 1%.

Q: Why have you chosen to use inertia mode rather than load bearing mode when testing?
First a little background on the 4 modes our SuperFlow SF840 is capable of:

• "simple inertia" is an inertia based mode that uses the mass of the roller to simulate load; this mode is best for acceleration testing (non adjustable load)
• "steady state control" is an inertia based mode that requires a variable load; this mode is primarily used for tuning purposes
• "controlled acceleration" is a load bearing based mode that requires a variable load; this mode is primarily used for finding maximum power numbers
• "road load" is a load bearing based mode; this mode is primarily used for simulation and diagnostic purposes

The Mods Face Off tests have always been in inertia mode and have used either simple or controlled acceleration. It was determined very early on that the performance numbers in the controlled acceleration tests were "close enough" (1-2hp difference?) to those found with simple inertia mode. It was decided at that point to standardize on simple inertia as it causes much less stress on the vehicle, and is more informative for seeing "up the curve" improvements.

Load bearing mode was briefly considered, but keeping in mind these are, after all, our daily drivers the higher numbers we were sure to see (around 3hp more per pull) simply didn't justify the exponentially more stressful test on the car; and now that we're over 200+ pulls we all think this was the right decision. The negative of this decision was the 2 gear pull manifestation, rather than (along with other computer tricks that would have been required) the single gear pull ability of load bearing mode. However remember, on that point, with this mode we are able to see true "up the curve" improvements in torque and horsepower. Load bearing would have shown us this effect only minimally.

Q: Why do all your released graphs show MPH for the X axis, rather than RPM like is traditionally shown?
Simple; if we tried to plot it in terms of RPM, the graphs would look like this:


What you're looking at is a two gear pull. Of course, since the transition from the first gear to the second gear drops the RPMs, we end up with the double back you see above. We are working on a tool that will allow us to further define the raw data, and isolate a single gear, but I can't promise when that will be available.

Q: Why the heck do you make 2 gear pulls anyway? Isn't the traditional way a single gear pull?
This 2 gear pull standardization we've lived with since day one is a direct result of the wonderful computers in our 5.7 LXs. They, without significant reprogramming and/or fuse manipulation simply wont let us stick in a gear under power load. They will ALWAYS drop down a gear when WOT is initiated and the RPMs will allow. Of course those RPMs are right where we want to start the pull.

This is not entirely a bad thing however. Due to this 2 gear pull, we've been able to compare shift patterns of various mods. We've also been able to draw "up the curve" or acceleration comparisons; something that would be far less pronounced on single gear runs.

And, ultimately, while a single gear pull (in load bearing mode) is bound to yield larger numbers (and a far cleaner graph), we don't necessarily care! Remember, our goal is to find relative differences in modifications, not global numbers for a given mod. In other words, if a vehicle with mod A shows 100hp and that same vehicle with mod B show 110hp under identical conditions and testing methods, we can safely say that mod B has outperformed mod A by 10%. This ratio will hold true for a single gear pull as well.

Q: Why do you emphasize average results rather than the maximum result for any give mod?
Averages tend to minimize anomalies. Experience with these LXs have taught us that they can act quite differently in back to back runs. The only true way we can see of comparing relative results given this known variability is to select a sample size, and average based on that. We always do publish each and every run's results, so you yourself can see the maximums, but when it comes to conclusions we make, they will always be based on the average of a set of dyno pulls.

[CoolVanilla]

Q: How to you deal with "heat soak"?
Experience, careful data analysis and extensive discussion has taught us that heat soak is a real problem. Heat soak is when the engine heats to a point where it is no longer producing optimum horsepower. Heat soak can really skew dyno numbers; not to mention hurt you're times at the track! However heat soak is also a reality that we have to allow. When you drive down the street the hood is down, and so it is on MFO dyno runs. 5-foot fans in the front and rear of the car simulate a 45-mph speed, and whatever heat is built up under the hood during the dyno run ... is what it is. Just like it would be on your car, driving down the road.

We've determined, via no small effort of trail and error, that there seems to be a sweet spot for our setup. If we make a run, turn off the vehicle, pop the hood, keep the cooling fans blowing and wait a full five minutes, heat soak effects are almost null. Obviously, waiting longer will take what little issue it is at 5 minutes and reduce it to zero, but we are bound by a strict time schedule when on the dyno and 5 minutes is just about as long as we can afford to wait between pulls. Lucky for us, it seems to be the point of diminishing returns.

Q: Why have you chosen 3 pulls as your sample set? If you're averaging, and you know these cars can act differently from run to run, shouldn't you be making more pulls in order to truly form an average?
This is where reality collides head on with theory; in theory, this question's answer is "You're 100% right; the more pulls the better!" In reality however, the answer is "Well, finances and time dictate 3 pulls, no more."

Here is the financial reality of our dyno time: each pull on the dyno costs us a minimum of 6 minutes. We are paying right around $1.25 per minute that we have the dyno dedicated to our tests, whether the dyno is rolling or not (and that is one hell of a bargain). So, if we consider the last round of testing had 14 different modification combos, across 2 different cars we can figure that every pull we make costs us: 6mins * 14mods/combos * 2 cars + 15 minutes of fudge = right around $220 per pull! 3 pulls costs us $660. 4 would cost us $880 ect... and, it just so happens that 3 pulls with 2 cars and 14 mods is over 8 hours, which means we are into a second day of testing.

The reality of our situation dictates 3 pulls. And, when we did look into more than 3, we found the averages were "close enough" to not justify the extra expense and time. And again, since we are all about the relative changes, as long as the 3 pulls per mod stays consistent, we can draw entirely valid conclusions from them as the variabilities will simply wash out.

Q: Ok, I get the dyno stuff. But what about real world results? Do you guys also figure what a mod is worth at the track?
Each and every MFO event consists of two parts. (1) the dyno tests and (2) the track tests. In our opinion, one component without the other makes for an incomplete test.

We always choose a day when both the dyno and track are available to us, thus ensuring (as best as possible) consistent weather conditions. After the day at the dyno is complete, we caravan over to the track and run the mods tested earlier in the day, late into the night. Due to time constrains (and install abilities) we are often required to selectively pick mods to run that evening. We try to pick the best performers on the dyno from earlier in the day. If time permits, we also try to run the worst performing mod as a sanity check.

Just as controlled variables are critical to consistent dyno results, so are controlled circumstances at the track. To that end, we always ensure the same conditions at the track, as best we can. Our results are always found at the same track (Sacramento Raceway), by the same driver (the talented gotls1), with the same fuel levels, ect...

When reviewing a MFO result, you may not see track numbers for that mod. We'll typically try to make a note as to why; either stating we did not test it at the track (and reviewing the dyno numbers will usually be a clue as to why), or if we did test it that it made zero distinct difference. Even in that case however, I still try to post up the track times.

Q: You've mentioned "calibrating the dyno". What exactly does that mean?
This particular dyno make is unique in that it measures force and acceleration, rather than one and calculating either. So, essentially we are using two methods of measurement and thus each requires calibration.

When I was there last, I arrived early enough to catch them calibrating the strain (force) gauge. Essentially what they do is cantilever 100lbs out from the gauge so that it measures an actual 500lbs. They then take a sample and adjust the read value to the known actual value of 500lbs. They then remove the weights and calibrate for 0lbs. They repeat this procedure until the read values match the known values exactly. On this particular day, the strain gauge was off by less than 3lbs before calibration (over a month of daily use). Not bad at all.







As for acceleration; this is measured by an encoder on the shaft itself. Basically its measuring how fast the drums are spinning. Then by sampling very quickly it calculates a rate of change and thus acceleration. This type of detection is touchless and therefore only rarely requires calibration.

Q: What does this dyno look like?
Here are a few pics of the internals of this SuperFlow.






Those rollers are the biggest available at 45" diameter. Why so large? They provide a larger contact patch to the tire thus mimicking truer to life conditions (ie more rubber on the road). This is also a safety feature in that the sidewalls of our tires aren't stressed more than usual (smaller contact point = the more focused the stress is on that area of the tire.)



Yep, thats a rear out of a dump truck!



Next time I'll have them pull up those back plates so we can see the eddy current generators and other internals.

[CoolVanilla]

Q: How do you make a pull and what settings do you use?
When someone asks what "settings" we use, I have to refer them back to our type of dyno we have and the way we use it. We use "inertia" mode and we measure both force and acceleration. This means we don't need to calculate anything, and therefore we have no weight settings or other variables that another dyno might need. We do specify a total tire diameter, but this is just so the speedo matches. It is not a value used in calculating power numbers.

In terms of how to run an actual dyno pull:
If we plan to run a multi gear test, we test 2nd and 3rd gears. We'll start out at a roll of around 30mph in 2nd (this is the gear we've determined the TCM/PCM will hold without dropping into 1st on a WOT) then we hammer home all the way thru the shift into 4th. This provide a great way to see acceleration differentials as well as how we are shifting.

If we plan to make a single gear pull, we will roll thru 1-3 and into 4th at about 65mph. We then hammer home until we RPM out. Be warned on this though; if you've removed your speed limiter you WILL get up into the 180mph range before you RPM out.

In all cases in all tests that make it into the event results, THE HOOD IS ALWAYS CLOSED. If you see a picture with an open hood, its not during a dyno pull or its not an official test run (and since dyno time is both expensive and paid for dearly, we don't waste many pulls running things that aren't expressly part of the test protocol).

I have shown these dyno tests to various people who have dyno shops and they all laughed at how pathetic the tests were. They are not even close to real world testing. How do you respond to that?
My first thought is, lets get these folks on the phone. Maybe they have a viewpoint we haven't been introduced to yet, that could help us derive a more informative and accurate test next time!

If the conversation turns to dyno technicalities and intricacies I just don't have the brain for, I then defer to our dyno's operator, Bob Crespo of SVS R&D. He has always made is clear he is willing to take a phone call and discuss such things. I have found him to have an extremely open mind; he'll be the first to admit mistake or consider an alternative method. But, having said that, come ready to back up your claims. If you don't want to talk on the most extreme of technical levels, don't pick up that phone.

Should anyone wish to discuss the technical details of dyno operation, Bob is your man. Should anyone have any other concerns related to these tests, I'm your guy...methodology, procedure, execution, planning, result formulation, information presentation... I'm the dude that you need to yell at if you find something you don't like. Bob, essentially does what we ask but with an educated and experienced eye that wont let us blow our rigs or test something in a way that yields useless results.

All we both ask is that you identify yourselves and state your calling/PMing in reference to the MFO tests. I also ask that you keep in mind Bob has a company to run and that the MFO tests are in no way affiliated with him (we're just customers of his!). Please be patient should he not have the time to discuss your concerns right then and there, and please don't abuse the kindness he is extending us.

[CoolVanilla]

(last one)

[RobAGD]

(reserved)

Heck I wanted one too

-R

[MattRobertson]

(reserved)
what the hell... :-)

[Guest]

{{{{{ reserved in stereo }}}}}



Buncha smartazzes we are...

[crhemi]

(RESERVED)
I better get mine too!

[boggart]

Interesting, very interesting.

(reserved for editing)

[Rev.Hammer]

POSSIBLY... but this interest needs validation and verification!

[Big Boy]

Very informative.....tell me more..............

[CoolVanilla]

The first pass at the Q&A is up. Please, ask your methodology questions here! This is the place to discuss any and all concerns you may have about what we do. Its constructive discussions like this that help make our good tests even better.

[Husker]

<reserved>
ok, thanks for the info CV

[Ozzie]

With all the back and forth regarding methodology on a related thread, I thought I'd throw this in for the mulling thereof....

...critical-thinking skills are necessary for engaging in critical thinking, but they are not sufficient. You also have to want to think critically. If you have good critical-thinking skills but for some reason are not motivated to deploy them, you will reach conclusions and make decisions no more rationally than someone without those skills......

read the full article here....

http://online.wsj.com/article/SB1161...lance_most_pop

[CoolVanilla]

With all the back and forth regarding methodology on a related thread, I thought I'd throw this in for the mulling thereof....

...critical-thinking skills are necessary for engaging in critical thinking, but they are not sufficient. You also have to want to think critically. If you have good critical-thinking skills but for some reason are not motivated to deploy them, you will reach conclusions and make decisions no more rationally than someone without those skills......

read the full article here....

http://online.wsj.com/article/SB1161...lance_most_pop

So true Ozzie. So true.

Bring on the questions guys! This is THE place to air any thoughts/feelings/concerns/questions/comments/insults (ok so long as you stay nice about it). Old discussions or new. Help us build the best Q&A we can, and help make these tests better! We most certainly do not know it all and if you can show us something we're missing, do so!