How is σ within calculated?

From R-bar / d2 in a subgrouped study, or moving range / 1.128 in an I-MR chart.

Capability Conversion

Process Capability Formulas

3 min read Last updated July 2026

The capability formulas are simple; the discipline is in choosing the right sigma and confirming the process is stable and normal.

Open the Process Capability Calculator Quality Statistics Knowledge Center

The engineering question this page answers

Which formula is the correct one to use for this dataset, this specification, and this decision?

Decision logic

Confirm data type: variables (continuous) is required for Cp/Cpk/Pp/Ppk ↓ Confirm normality and stability ↓ Select σ estimator: within (R̄/d₂ or S̄/c₄) for Cp/Cpk; overall S for Pp/Ppk ↓ Two-sided spec → Cp; nearest-tail → Cpk ↓ Report the exact formula and σ definition used

Capability study readiness

Formula

Cp = (USL − LSL) / (6 σ_within) Cpk = min( (USL − x̄) / (3 σ_within), (x̄ − LSL) / (3 σ_within) ) Pp = (USL − LSL) / (6 σ_overall) Ppk = min( (USL − x̄) / (3 σ_overall), (x̄ − LSL) / (3 σ_overall) ) Cpm = (USL − LSL) / (6 × sqrt(σ² + (x̄ − T)²)) Cmk = min( (USL − x̄), (x̄ − LSL) ) / (3 σ_machine) σ_within = R̄ / d₂ (subgroup ranges) = S̄ / c₄ (subgroup standard deviations) σ_overall = sample standard deviation of all data Z_nearest = 3 × Cpk

Examples

Example. Aluminium bracket length. USL = 100.05, LSL = 99.95, x̄ = 100.00, σ_within = 0.010, σ_overall = 0.014, target T = 100.00.

Metric	Value	Interpretation
Cp	1.67	Entitlement excellent
Cpk	1.67	Centered; capability = entitlement
Pp	1.19	Long-term inflated by drift
Ppk	1.19	Customer-facing capability
Cpm	1.67	On target
Ppk / Cpk	0.71	29% drift — investigate before PPAP

Conversion conditions

Distribution modelNormal only when justified; otherwise use a documented non-normal method.

Specification modelcontext-dependent

Tail conventioncontext-dependent

Centering assumptionCp/Pp use spread; Cpk/Ppk use mean distance to the nearest limit.

Sigma basiscontext-dependent

Sigma-shift conventionnot-applicable

Output typeindex-comparison

Required inputs

LSL and USL where applicable
mean
within sigma
overall sigma
stability and distribution evidence

Invertibility. Formula selection follows the specification model and the chosen within or overall sigma.

Limitations

Formula output is not trustworthy for unstable data or an unjustified normal model.

Worked example and engineering limits

Worked example. For LSL 4, USL 16, mean 9, within sigma 1.0 and overall sigma 1.2: Cp 2.00, Cpk 1.67, Pp 1.67 and Ppk 1.39.

Failure case. Applying normal Cp/Cpk formulas to unstable or non-normal data without justification creates a misleading index.

What cannot be inferred. Conformance history or customer acceptance from an index alone.

Engineering next step. Choose the index and sigma basis from the study purpose, then document assumptions alongside the result.

Common mistakes

Mixing σ definitions inside a single report.
Using Cp on a one-sided specification — undefined; use Cpu or Cpl.
Applying Cpm without a defined engineering target.
Using Cmk (machine capability) as if it were process capability — Cmk excludes operator, material and environment.
Publishing indices without stating the σ estimator and the sample size.

Engineering insight

There is no single "process capability formula". There is a small family, each with a defined purpose. Every capability report should name the formula, the σ estimator, and the sample basis. Anything else is a number without a decision behind it.

When NOT to use this metric

Attribute data — use p/np/u/c charts and defect-rate capability.
Non-normal continuous data — use Johnson/Box-Cox transformation or percentile-based capability (ISO 22514-2).
Unstable or out-of-control processes — indices are undefined.
Sample sizes below the minimum for the chosen estimator.

Relationship to other capability metrics

Cp / Cpk / Pp / Ppk: Same design intent, four different σ / centering combinations.
Cpm: Penalises deviation from target — required when the target is not the spec midpoint.
Cmk: Machine-only capability — a pre-requisite for process capability, not a substitute.
Z, PPM: Derived from Cpk under normality — not primary formulas.

Recommended actions

Use the free Process Capability Calculator to compute Cp, Cpk, Pp, Ppk, sigma level, and predicted PPM from your measurement data — no sign-up required.

Continue the investigation

Interpret results via Process Capability Index, plan the study in Process Capability Study, and analyse with Process Capability Analysis. Use the Process Capability Calculator to verify by dataset.

Verification checklist

Variables data, normal, stable
σ estimator explicitly named
Formula name matches the specification type (one-sided vs two-sided)
Target used only when Cpm is reported
Confidence interval or sample size disclosed

Assumptions and applicability

Process condition: statistical stability is required.
Distribution assumption: approximately normal data.
PPM convention: nearest specification-limit tail.
Sigma-shift convention: no sigma shift applied.
Cpk alone identifies distance to the nearest specification limit; it does not determine total two-sided defects unless centering or both tail distances are known.
Predicted PPM is a model estimate, not a replacement for observed defect data.