Free tool · runs in your browser

Question sample size: can you trust your visibility rate?

A rate measured on only a few questions can be far off: the true value could be anywhere in a wide range. Enter your numbers to see that range, and how many questions it takes to narrow it. Nothing you type leaves this page.

Different questions, each asked once per check.

1 for one check. Four weekly checks of the same questions = 4.

Counted across all 20 answers (questions × checks).

10 means your rate is right to within 10 points either way. From 1 to 49.

Your rate

Rate

20.0%

4 of 20 answers

Likely range (95%)

8.1% – 41.6%

±16.8 points · 33.5 points wide

Fewer than 32 different questions: this rate can still be far off. The true value could be anywhere in a range wider than a third of the scale, so don't read much into it yet. Tracking more questions narrows it fastest.

Questions needed for a margin

The smallest number of questions, one answer each, whose 95% range is no wider than the margin. 50% is the worst case: plan on that column if you do not know your rate yet.

Margin (points)At 50% (worst case)At your rate (20.0%)
±5381245
±109360
±153926
±202114

With your 20 questions checked once, the margin at this rate is ±16.8 points.

Method: Wilson score interval, 95%, without continuity correction, the same one the Promvia dashboard prints. ±X points is half the interval's width. The interval covers sampling noise only: it cannot correct a question set that leans toward questions you already do well on.

Read the guide: how many questions do you need? → How we measure →

How the question sample size calculator works

Maintained by the Promvia team · Last updated

It answers two questions about a visibility rate measured on a set of questions: how wide the range of true rates behind it is, and how many questions a narrower range takes. Both come from the 95% Wilson score interval, computed in your browser with the functions the Promvia dashboard prints its intervals with. Nothing you type is sent to a server.

What it calculates

  • The rate: answers that named you, divided by every answer collected, which is questions × checks pooled.
  • The 95% interval around that rate: the Wilson score interval with z = 1.96, without continuity correction. It stays between 0% and 100%, and it does not shrink to a single point at 0 of n or n of n, where the textbook p ± z·√(p(1−p)/n) does.
  • The margin: half the interval's width, in percentage points. The Wilson interval is not symmetric around the rate, so both bounds are printed as well.
  • Questions needed for a margin: for ±5, 10, 15, and 20 points and your own target (1 to 49), the smallest whole number of questions, one answer each, whose interval at that rate is no wider than ± the margin. It is solved in closed form, then checked against the whole numbers on either side.
  • With more than one check pooled, a second interval: the same rate measured on the questions alone, as if every repeat check copied the first.

How to read the result

The calculator opens on 4 of 20: a rate of 20.0% and a 95% interval of 8.1% to 41.6%, which is ±16.8 points, 33.5 points wide.

An interval wider than 33 points, a third of the scale, cannot say which third of 0–100% the rate is in. Below 32 questions the interval at a 50% rate is that wide, so every rate below that size is marked as based on too few questions to trust whatever it came out at, and two rates on the same number of questions are always marked alike. The opening example is one, so under it the calculator prints: “Fewer than 32 different questions: this rate can still be far off. The true value could be anywhere in a range wider than a third of the scale, so don't read much into it yet. Tracking more questions narrows it fastest.”

The rule is set on n, not on the interval a check happened to produce. 0 of 8 gives 0–32% and 2 of 10 gives 6–51%: a rate near 0% or 100% is measured more precisely than one near 50%, but the next check on the same number of questions can land near 50%, so both are marked.

In the questions-needed table the 50% column is the worst case: the interval is widest at 50%, so a count read from that column holds at any rate. At 50%, ±10 points takes 93 questions and ±5 points takes 381.

Limits

  • Sampling noise only. A question set that leans toward questions you already do well on gives a narrow interval around a rate that does not describe your market, and no sample size corrects that.
  • Every answer is treated as an independent trial. Repeat checks of the same question are correlated, so the pooled interval is narrower than the evidence supports; the questions-only interval is the other end, and the true uncertainty lies between the two.
  • Answers from different AI assistants to the same question are correlated in the same way: counting each assistant's answer as its own trial narrows the interval by more than the evidence supports.
  • One rate at a time. Whether two rates differ, for example between two periods, takes an interval for the difference, which this calculator does not compute: two overlapping intervals do not by themselves show that the rate stayed the same.
  • Whole numbers only, and a target margin from 1 to 49 points.

Questions about this tool

How many questions should I track?

As many as the margin you would act on requires. At the worst case of 50%, ±10 points takes 93 questions and ±5 points takes 381, one answer each. The guide linked above works through the choice for a real tracking plan.

Do repeat checks count as more questions?

Only in part. Answers to the same question are correlated, so 20 questions checked 4 times are not 80 independent answers. With 16 of 80 the pooled interval is 12.7% to 30.0%; on the 20 questions alone it is 8.1% to 41.6%. The calculator prints both, and the true uncertainty lies between them.

What does Wilson do that the textbook formula does not?

On small counts, such as tens of questions, the textbook p ± z·√(p(1−p)/n) drops below 0% for small numerators and collapses to a single point at 0 of n: on 0 of 10 it prints 0 to 0%. Wilson gives 0–28% for the same counts, and it is the interval the Promvia dashboard prints.

Is what I type stored or sent anywhere?

No. The page computes in your browser and makes no request with the numbers you type.

When does the calculator say a rate is based on too few questions to trust?

When it was measured with fewer than 32 questions: at that size a 95% interval can be more than 33 points wide (it is, at a 50% rate). The rule reads the number of questions alone, so it never marks one rate and spares its neighbour on the same sample. The same rule marks rates in Promvia's weekly summary email, CSV export and API, and on the AI Source Index AI assistant pages.

Related tools and guides