How to compare AI tools fairly.
A comparison is only as useful as the task, evidence, and limits behind it. You do not need a laboratory to run a better test—but you do need to make the conditions clear.
1. Start with one real task
Choose a task you genuinely need to complete, such as turning lecture notes into a revision sheet, summarizing a policy, or drafting a project update. Define what a successful result must contain before testing any tools.
- What input will each tool receive?
- What must the output include—and what must it avoid?
- Which errors would make the result unusable?
- How much review or correction would be acceptable?
2. Keep the test conditions as consistent as possible
Use the same source material and the same task instructions for each system. Record any unavoidable differences, including available features, settings, file limits, or account tiers. If the systems use different model versions, document that instead of describing the comparison as perfectly controlled.
Save the exact prompt and source file. If you change the prompt after seeing an output, mark that as a new run; otherwise, it becomes difficult to tell whether the difference came from the model or the instructions.
3. Choose criteria before looking at the results
A short rubric helps prevent a polished or confident-sounding answer from automatically winning. For source-based tasks, useful criteria may include:
| Criterion | What to inspect |
|---|---|
| Source fidelity | Are claims faithful to the provided source? Are unsupported additions clearly marked? |
| Completeness | Are the required sections, definitions, examples, and important details retained? |
| Clarity | Can the intended reader understand and use the result? |
| Organization | Is the structure consistent and easy to navigate? |
| Revision effort | How much checking and editing is needed before the output is usable? |
Define what each score means. For example, write down what earns a 1, 3, or 5 before scoring. If you calculate a weighted total, publish the weights and formula so another person can reproduce the arithmetic.
4. Preserve raw outputs and identify them carefully
Keep the original responses unchanged. Store cleaned-up versions separately and label them as edited copies. Record the provider, model or version shown in the interface, date and time when available, exact prompt, source material, and any settings that may affect the result.
If you anonymize outputs for blind evaluation, keep a private mapping record. Do not assign model identities from writing style alone. If the mapping cannot be verified, report the outputs as anonymous.
5. Separate observations from conclusions
A single prompt can show how systems responded to that task on that occasion. By itself, it cannot establish which model is universally best, prove a general learning benefit, or demonstrate a statistically reliable difference.
- Describe what the output actually did.
- Distinguish evaluator judgment from measured performance.
- Report missing records, ambiguous instructions, and possible setup errors.
- Avoid generalizing beyond the tasks and versions you tested.
6. Make the result reproducible enough to inspect
Publish the task definition, rubric, scoring calculation, representative outputs where permitted, and limitations. If some details are unavailable, say so. A transparent incomplete record is more useful than a complete-looking report that fills gaps with assumptions.
A compact checklist
- One defined task and clear success criteria
- Same source and prompt wherever possible
- Model/version and run details recorded when available
- Predefined rubric and transparent score calculation
- Raw outputs preserved and identities verified
- Limitations and unresolved questions disclosed
Use CurioLab’s AI tool comparison guide for a reusable scorecard, and visit the Experiment #001 results note to see why provisional scores, anonymous output labels, and missing execution records need to be treated carefully.