AMSTAR 2 vs ROBIS: which tool for appraising systematic reviews?

The short answer

Side by side

Quality and bias are different questions

A review can follow good practice on every item and still reach a biased conclusion, if it interprets its findings without regard to the limitations of the included studies. Equally, a review with several process weaknesses can still land on the right answer. AMSTAR 2 counts weaknesses in conduct and reporting and turns them into a confidence rating. ROBIS asks, for each weakness, whether it could actually have distorted the result, and then in phase 3 whether the review authors took it into account when they interpreted their findings.

AMSTAR 2 is not blind to interpretation. Its thirteenth item asks whether risk of bias in the primary studies was accounted for when interpreting the results, and it is one of the seven critical items. But in ROBIS the appropriateness of the conclusions is a full phase of the assessment, and the relevance question from phase 1 feeds into it. If your purpose is to decide whether to act on a review, that phase is what you are paying for.

Scope: what each tool was written for

AMSTAR 2 is written for reviews of healthcare interventions. Its items about randomized and non-randomized designs, meta-analytical methods and the funding of included studies assume that context. Applied to a review of diagnostic accuracy or prognostic factors, several items do not fit, and adapting them silently produces a rating that readers will misread as a standard AMSTAR 2 result.

ROBIS was built to apply across question types. That flexibility is why guideline programmes with mixed evidence often reach for it, and why some overview authors find it vaguer than AMSTAR 2 for a pure intervention question.

The critical-item mechanism in AMSTAR 2

AMSTAR 2's rating rules are severe by design. Any one critical flaw caps the review at Low confidence, and two cap it at Critically Low, regardless of how many other items were satisfied. In published overviews the large majority of reviews land in those two categories, most often because of a missing protocol or an incomplete list of excluded studies.

The developers allow assessors to adjust which items are treated as critical for a particular field, but only if the choice is made in advance and reported. Changing the critical set after seeing the ratings is the AMSTAR 2 equivalent of outcome switching.

What the empirical comparisons found

Head-to-head studies find the two tools closely related rather than identical. Lorenz et al. (2019) reported high concordance between overall AMSTAR 2 and ROBIS ratings, with inter-rater reliability moderate for AMSTAR 2 and fair for ROBIS. Perry et al. (2021) applied both to the same 31 reviews and found identical median agreement between raters, with AMSTAR 2 more straightforward to use and ROBIS easier to apply to reviews that included a meta-analysis. Buhn et al. (2017) found ROBIS had fair reliability and good construct validity, and that reliability depended on the raters' experience.

The practical lesson is the same for either tool: pilot on a handful of reviews, write down how you will handle the items that caused disagreement, and use two independent assessors with a reconciliation step. Neither tool is reliable in the hands of a single untrained reviewer.

Using both, and what neither does

Applying both tools to the same reviews is common in methods research and occasionally in overviews. If you do, present both sets of results and decide in advance how a discordant verdict will be reported; do not merge them into a single rating. If you only have capacity for one, choose it on scope and purpose, state the choice in the protocol, and apply it to every included review.

Neither tool is a reporting checklist: PRISMA tells authors what to report and should not be used to appraise a review. And neither rates the certainty of the evidence: that is GRADE, applied to the body of evidence within a review, not to the review as a document.