LogFAQs > #891236241

Giblet_Enjoyer posted...

Corrupt_Power posted...

Literally arguing that testing browsers after the average user has bogged them down with several years of garbage is better than testing them fresh. You've got good jokes, I'll give you that.

That's the average scenario, so yes.

Testing programs under non-real-world conditions is interesting in an academic sense but it's not actually relevant to anyone aside from academics

My point though is that testing against old, "lived-in" installs would give just as subjective results as if you tested the 2 browsers on random separate computers. The data is completely useless outside of that precise test case. Moreover, the data from both browsers don't really relate in a fair way to each other.

Let me give you an example of the kind of issues a lived-in install can have:
- Software version 1 has setting A, and over the course of that version's life, people are recommended to turn that setting on for compatibility with legacy systems, or maybe it defaults to enabled.
- Software version 2 adds optimization Z that dramatically improves performance, but cannot function while setting A is enabled, so it is disabled in that case.
- Legacy systems are still in use, albeit rarely, so the developers cannot remove setting A.
- The majority of users do not benefit from optimization Z because they have setting A enabled, even though they rarely if ever interact with legacy systems.
- A clean install of version 2 defaults setting A to disabled.

This is a massively simplified version of the kind of issues I'm talking about - a lot of the time the leftover settings are internal flags and other things that even your average savvy user might not know how to modify.
---

Posted with GameRaven 3.3