February 24, 2024

Compare Tools With a Test You Write Yourself

Demonstrations are designed to succeed. If you are choosing between drafting tools, the fairest comparison is a test you write from your own work.

Build the test from real tasks

Select five to eight tasks the tool would do in ordinary use: a routine proposal section, a summary of a long request document, an answer to a security questionnaire item. Use material you are comfortable sharing, and remove client details.

Include one task that is hard on purpose, such as a long source document with a few contradictory details. Easy tasks make all tools look alike. A difficult one shows where they differ, and how visible their mistakes are to a reviewer who is not looking for them.

Fix the instructions in advance

Give each tool the same instructions and the same source material. Write them down before the test starts. Changing the wording halfway makes the results impossible to compare.

Run each instruction more than once if the tool allows it. Results can vary from one run to the next, and a single lucky output can mislead you. Note the range, not just the best result, and prefer the tool that behaves consistently.

Decide the scoring in advance

Agree on a short set of criteria: accuracy against the source, usefulness as a first draft, tone, and how much editing was needed. Use a simple scale, and have two people score independently before they compare.

Resist the urge to score on style alone. A polished paragraph with a wrong fact is worse than a plain one that is right. Weight accuracy more heavily than tone, and say so in the scoring guide before anyone sees the outputs.

Include a trap or two

Ask about something not in the supplied material and see whether the tool admits it does not know or invents an answer. This tells you a great deal about how much checking the tool will need.

Note also how the tool signals uncertainty. A good tool will say when the source does not support an answer, and a poor one will fill the gap with plausible text. Count the invented answers, since each one is a possible error in a client document.

Look beyond the output

Ask where data is stored, who can see it, how long it is kept, and whether it is used to improve the vendor's product. The best draft is no use if the terms rule it out.

Compare the costs in the same terms as the benefits. Include licences, set-up, training time and the hours spent reviewing. A tool that saves less time but carries lighter data terms may be the better choice for a firm handling sensitive client material.

Record the result and revisit it

Products change frequently. Keep the test, the scores and the date, so you can run it again in six months and see whether your choice still holds.

Share the test with the vendors afterwards if they ask, and note their responses. A vendor who takes your findings seriously and explains what changed is showing you how they will behave after the sale, which is worth as much as any feature.

Keep the test simple enough to repeat

A test that takes a week to run will not be repeated. Aim for an afternoon of work by two people. The purpose is a reasonable comparison, not scientific proof, and a modest test that is actually rerun teaches more than an elaborate one that is not.

Keep the source material, instructions and scoring sheets in one folder, with a note on how to run them. When a new tool appears, or when your needs change, the next person can pick it up and have a comparison within a day. Whatever you choose, record why you chose it, so that the reasoning survives changes of staff and can be tested again when the products change.

VIPMarketing Editorial