Attribute Agreement Analysis: MSA for Pass/Fail Inspections

Attribute agreement analysis is a measurement systems analysis (MSA) method that tests whether appraisers making pass/fail or other categorical judgments agree with themselves, with each other, and with a known reference standard. It is the attribute-data counterpart of the Gauge R&R study used for continuous measurements. Visual checks, go/no-go gauges, and cosmetic grading all depend on human judgment, and that judgment can drift and fail just like a worn caliper. Before you trust pass/fail data enough to act on it, you need evidence that the inspection itself is repeatable and correct.

Why pass/fail inspections need their own MSA

A variable gauge study expresses measurement error in the same units as the characteristic. Attribute inspection produces categories, not numbers, so repeatability and reproducibility must be expressed as agreement rates instead. This matters because a leaky inspection quietly corrupts every downstream decision. SPC charts built on defect counts, scrap reporting, and lot acceptance all inherit the error. An inspection with a 10 percent miss rate ships one defective unit in ten with a passing record attached.

The three questions the study answers

• Within-appraiser agreement (repeatability): does the same inspector rate the same part identically on repeated blind trials? Low values point to an unclear or ambiguous defect definition.
• Between-appraiser agreement (reproducibility): do different inspectors rate the same part the same way? Low values usually mean each inspector has a private interpretation of the spec.
• Appraiser versus standard (effectiveness): do the ratings match the known truth? Two failure modes matter: the miss rate (defective parts accepted, the customer-facing risk) and the false alarm rate (good parts rejected, inflating scrap and rework).

How to design the study

1. Select 30 to 50 parts. Deliberately include roughly one third clearly good, one third clearly defective, and one third borderline. A study made only of obvious parts proves nothing.
2. Establish the reference standard. A quality engineer or expert panel classifies every part against the written CTQ definition, using lab measurement where possible; keep this truth set hidden from appraisers.
3. Run 3 appraisers through 2 or 3 trials each. Randomize part order every trial and blind the appraisers to part identity and to each other's results.
4. Use production conditions: the same lighting, station, tools, and time pressure as a normal shift.

Worked example: three appraisers, 30 parts, two trials

A machining cell runs an attribute agreement analysis on a visual burr inspection: 20 conforming and 10 defective reference parts, three appraisers, each rating all 30 parts twice, blind and in randomized order. The results:

• Within-appraiser: Appraiser A matched their own two trials on 28 of 30 parts (93.3 percent), Appraiser B on 26 of 30 (86.7 percent), Appraiser C on 29 of 30 (96.7 percent).
• Appraiser versus standard: counting only parts rated correctly in both trials, A scored 27 of 30 (90 percent), B scored 25 of 30 (83.3 percent), C scored 28 of 30 (93.3 percent).
• Between appraisers: all three agreed with each other across every trial on 22 of 30 parts (73.3 percent), and all three also matched the standard on 21 of 30 (70 percent).

Digging into Appraiser B: across 20 decisions on defective parts (10 parts, 2 trials), B accepted 3, a miss rate of 15 percent. Across 40 decisions on good parts, B rejected 3, a false alarm rate of 7.5 percent. B's overall effectiveness is 54 of 60 decisions, or 90 percent, which looks acceptable in isolation. The 15 percent miss rate tells the real story, and the 70 percent team-versus-standard score shows the system, not one person, needs work.

Kappa: agreement beyond chance

Percent agreement flatters the system because two people flipping coins would still agree sometimes. The kappa statistic corrects for that: kappa = (Po minus Pe) divided by (1 minus Pe), where Po is observed agreement and Pe is chance agreement. For Appraiser B versus the standard, Po = 0.90. B accepted parts in 40 of 60 decisions while the standard says 40 of 60 are good, so Pe = (0.667 x 0.667) + (0.333 x 0.333) = 0.556. Kappa = (0.90 minus 0.556) / (1 minus 0.556) = 0.78. As a rule of thumb, kappa above 0.75 indicates good agreement, below 0.40 indicates poor agreement, and many automotive customers expect 0.90 or better for safety-related characteristics. B sits in the marginal zone.

Acceptance criteria and fixing a failing system

Widely used AIAG-style guidelines: effectiveness of 90 percent or more is acceptable and 80 to 90 percent is marginal; miss rate of 2 percent or less is acceptable and above 5 percent is unacceptable; false alarm rate of 5 percent or less is acceptable and above 10 percent is unacceptable. When a study fails, the fixes are usually procedural, not personal:

• Create boundary samples and photo standards that show the worst acceptable and best rejectable condition.
• Improve lighting, magnification, fixturing, and inspection time.
• Retrain against the visual standard, then re-run the study to verify.
• Automate the check where judgment cannot be made consistent, and record the required method in the control plan.

If the defect definition itself is disputed, run a structured DMAIC project rather than arguing at the inspection bench.

Where Fabrico fits

Attribute agreement analysis validates the judgment; Fabrico supplies the trustworthy production data around it. Fabrico delivers real-time OEE and production monitoring, including computer vision on machines with no PLC, so output and quality numbers reflect what actually happened on the line. Its field-ready CMMS handles the operational side of a healthy measurement system: recurring work orders for gauge calibration, scheduled MSA re-studies, boundary-sample reviews, and full asset history for every inspection station. Fabrico is EU-built with EU data residency, so the record stays auditable for European compliance requirements.

Frequently Asked Questions

How many parts and appraisers do I need for an attribute agreement analysis?

A common minimum is 30 parts, 3 appraisers, and 2 trials (180 decisions). Use 50 parts when defects are subtle. The sample must include borderline parts near the accept/reject boundary, where inspection systems actually fail.

What is an acceptable kappa value?

Kappa above 0.75 is generally considered good and below 0.40 poor. Many automotive and medical customers require 0.90 or higher for critical characteristics. Always read kappa alongside the miss rate, since a decent kappa can hide an unacceptable escape risk.

How often should the study be repeated?

Re-run the study when appraisers change, when the defect definition or boundary samples change, after a customer escape, and on a fixed cadence (annually is typical). Treat it like gauge calibration: an owner, a due date, and a record.

Want inspection results, scrap, and OEE flowing from one trustworthy real-time source? Book a Fabrico demo and see your quality data the way an auditor will.