Regulatory data quality and lineage: from source to report

Define the data in business terms

A field name is not a definition. “Outstanding balance” may refer to principal only, principal plus accrued interest or another measure. “Customer” may mean an individual account holder, a legal entity or a connected group. The reporting purpose determines which meaning is appropriate.

Maintain a business definition for critical fields, including scope, units, timing and relevant exclusions. Record the owner who can resolve interpretation questions. Technical mappings should implement those definitions rather than become their only explanation.

The Basel risk-data principles provide an international reference for effective aggregation and reporting. ^[1] Their formal scope should be distinguished from voluntary use of their ideas. For Indian institutions, the applicable RBI reporting directions provide the relevant domestic requirements. ^{[3] [4]}

Identify authoritative sources and responsibilities

For each important data element, identify the source system and the business owner responsible for its quality. One system may be authoritative for balances while another holds approved customer grouping or collateral information. A single “source of truth” slogan does not resolve those relationships.

Define handoffs between source owners and reporting teams. The source owner should understand the required population, period and attributes. The reporting team should know the source's limitations and how corrections will be communicated.

Keep ownership current as systems and teams change. A field without an accountable owner often becomes a recurring reporting adjustment. Assigning responsibility makes it possible to correct the cause rather than repeatedly repair the downstream output.

Map lineage at a useful level

Lineage should connect source fields, transformations, intermediate outputs and final report items. It does not need to display every technical operation to every user, but it should provide enough detail for a material figure to be traced and reproduced.

Record filters, joins, classifications, calculations and manual adjustments. These operations can change the population or meaning of data. A lineage diagram that shows only system names may conceal the step where an error was introduced.

Version the mapping and transformation logic. A current diagram cannot explain an earlier return if the process has changed. Retain the source version, rule version and output version together so the reporting history remains intelligible.

Test completeness separately from accuracy

Accuracy asks whether recorded values are correct. Completeness asks whether the required records and attributes are present. A dataset can contain perfectly accurate values for only part of the institution's business.

Use independent population controls where possible. Compare expected branches, products, accounts or transaction streams with what arrived. Row counts and financial totals can identify problems, but neither is sufficient alone. Duplicate records may offset missing records in a count, while value errors may cancel in a total.

RBI's supervisory-return directions address data governance, reconciliation and source records. ^{[3] [4]} A practical implementation translates those expectations into specific checks at the points where data enters and changes within the reporting process.

A worked example: a join that loses accounts

Suppose a hypothetical reporting process joins a loan table with a product-classification table. Several new product codes are absent from the classification table. The technical join excludes the affected loans, and the final report appears structurally complete.

A total reconciliation identifies a shortfall, while a population check isolates the missing product codes. The team corrects the mapping, reruns the process and reviews other outputs using the same join. It also changes the process so unknown codes create an exception rather than disappearing silently.

The example illustrates why lineage and quality checks work together. The reconciliation detects a difference; the lineage identifies where the population changed; source ownership makes the permanent correction possible.

Make quality rules meaningful

A quality rule should state the issue it detects and the consequence of failure. Examples include missing identifiers, inconsistent dates, invalid classifications or totals that fail to reconcile. Distinguish rules that block processing from warnings requiring review.

Avoid large rule catalogues with unclear purpose. A rule that produces thousands of ignored warnings can weaken attention to important exceptions. Review its design, relevance and the action expected from the owner.

Measure results by significance as well as count. One missing attribute affecting a large exposure may matter more than many harmless formatting differences. The institution needs a documented way to prioritise issues without hiding unresolved defects.

Control corrections at the right point

When a defect is identified, determine whether the source, transformation or output needs correction. Fixing the final report may be necessary for an immediate deadline, but it should not replace a permanent source or logic correction where one is required.

A manual adjustment should record the reason, amount, affected population, evidence, owner and approval. Identify whether it is temporary and what will remove the need for it. Repeated adjustments deserve management attention because they create dependence on individual knowledge.

Assess downstream impact. A corrected customer classification can affect several reports, capital calculations and management views. The lineage should help identify those dependencies so the institution does not fix one output while leaving related ones inconsistent.

Preserve reporting snapshots and reproducibility

A reporting snapshot should identify the source deliveries, transformation versions, assumptions and approved output used for a particular period. Preserve original data as required under the institution's retention framework, with appropriate access controls.

Reproducibility means another qualified person can follow the documented process and understand how the result arose. It does not require unrestricted access for every user, but it does require controlled availability of the necessary evidence.

Check that historical outputs remain interpretable after system changes. If a code table is overwritten with current values, an old report may no longer be reproducible. Effective dating and versioned reference data help preserve the meaning of earlier calculations.

Govern changes to critical data

New products, acquisitions and technology releases can alter data definitions or availability. Include reporting impact in change assessments. A business feature may work correctly while removing an attribute required for a regulatory return.

Test affected transformations and reconciliations before deployment. Use representative cases and compare expected results. Where a transition requires parallel processing, define the comparison criteria and the point at which the new process can become authoritative.

The Basel operational-risk principles provide background for controlled change and technology-related risk management. ^[2] Operationally, the institution needs a visible connection between the change approval and the reporting processes that depend on the changed data.

Prioritise a small set of critical data elements

Begin detailed governance with fields whose failure could materially affect reporting or decisions. Examples might include exposure amount, customer identifier, product classification, contractual due date and entity code. The actual selection should follow the institution's reports and risks.

For each critical element, document its definition, source, owner, transformations, quality rules and affected outputs. This creates a manageable starting point and demonstrates how governance works in practice. Extend coverage as the method becomes reliable.

Avoid designating every field as equally critical. An undifferentiated catalogue makes prioritisation difficult and can consume effort without resolving the most significant weaknesses. Reassess criticality when products, requirements or reporting dependencies change.

A useful exercise traces one material return item backwards to its critical inputs, then forwards to other outputs using those inputs. This reveals shared dependencies and helps estimate the impact of a defect. It also provides a concrete way for business and technology teams to agree where improvement effort will have the greatest benefit.

Turn data-quality reporting into remediation

Management reports should show material defects, ageing, affected outputs and progress on permanent corrections. A high pass rate can conceal a small number of serious unresolved issues. Explain the residual effect on reporting confidence.

Distinguish detection from resolution. A defect repeatedly identified by a control is not under control merely because the alert works. The institution needs an owner and a realistic plan to remove or appropriately manage the underlying problem.

Reliable regulatory data emerges from clear definitions, accountable sources, traceable transformations and controlled corrections. When those elements are connected, a reported figure becomes more than a number: it becomes an explainable result whose basis can be checked, challenged and reproduced.