Validation
Validation is the gate between staging and Commit. It runs hundreds of rules — data types, referential integrity, business sanity, regulatory limits — and produces an Exception Report. The Commit button is disabled until zero errors remain and every warning has been acknowledged.
At a glance — Path: MFI Suite -> Migration -> Runs -> open Run -> Run Validation. Group: MFI / Migration Lead to trigger, MFI / Data Steward to acknowledge warnings. Average runtime for a 25,000-member staging set: 2-6 minutes on the staging container.
Three classes of rules
Every validation rule belongs to one of three classes. The class determines who can override and how the rule appears in the Exception Report.
| Class | Severity | Examples | Override allowed? |
|---|---|---|---|
| Structural | Error (blocking) | Required field missing, foreign key not in staging or production, duplicate source_key, wrong data type | No — must be fixed at the source |
| Business | Warning (acknowledgeable) | Member older than 110 years, savings balance over KES 5M, loan outstanding > principal, schedule sums do not match outstanding | Yes — Data Steward acknowledges with comment |
| Regulatory | Warning (acknowledgeable, but auditor-visible) | Member has no KRA PIN but balance > KES 1M, member status active but no KYC documents, large transaction without GOAML log | Yes — but every acknowledgement is exported to the Cutover report |
Structural rules (the always-blocking ones)
Structural rules cannot be acknowledged or bypassed. They protect the integrity of the production model — if a Loan row has no matching Member, committing it would create an orphan.
- REQUIRED_MISSING — required target field is empty (e.g. member.full_name).
- TYPE_MISMATCH — value cannot be coerced to the target type (e.g. 'N/A' in a date field).
- DUP_SOURCE_KEY — same source_key on two staging rows within the same Run or across uncommitted Runs of the same Source.
- FK_NOT_FOUND — staging.savings.member_key has no match in staging.member nor in production res.partner.x_migration_source_key.
- ENUM_INVALID — value outside the allowed enum (e.g. status='X' when only active/dormant/closed/blacklisted are allowed).
- TRANSFORM_FAIL — the mapping transform raised an exception.
Business rules (the smart-warnings)
Business rules catch data that is technically valid but suspicious. The Data Steward reviews each warning and either fixes it or acknowledges with a free-text reason. The reason becomes part of the Mapping Audit trail forever.
| Rule code | Description | Typical cause in Kenya |
|---|---|---|
| AGE_OUTLIER | Member DOB implies age <18 or >100 | Loan Performer's '01/01/1900' placeholder for missing DOB |
| BALANCE_NEGATIVE_SAVINGS | Savings balance is negative | Overdrawn shares account, common in SACCOs that allowed advances on shares |
| LOAN_OUTSTANDING_GT_PRINCIPAL | Outstanding > original principal | Accrued interest never capitalised in legacy; needs Steward decision to either capitalise or split into principal+interest |
| SCHEDULE_SUM_MISMATCH | Sum of schedule.principal_due != loan.outstanding | Most common business issue — legacy schedules are stale because partial payments were posted as lump-sum txns |
| KRA_PIN_INVALID_FORMAT | KRA PIN does not match Axxxxxxxxx or Pxxxxxxxxx | Member typed it in lowercase or with spaces |
| PHONE_NOT_KENYAN | Phone does not start +254 after transform | Migrant worker with a Tanzanian number, or operator typo |
| DORMANT_WITH_ACTIVE_LOAN | Member status=dormant but has an active loan | Legacy did not reconcile status when a defaulter went silent |
| MEMBER_NO_PRODUCT | Member has no savings and no loan | Genuinely closed members never archived; usually safe to ignore |
Warning — AGE_OUTLIER and DORMANT_WITH_ACTIVE_LOAN look benign but if acknowledged carelessly they corrupt later regulatory reports (CBK Form B, SASRA aged loans). Treat acknowledgement as a deliberate, documented decision.
Regulatory rules (the auditor-visible ones)
Regulatory rules check compliance with Kenyan regulators — CBK, SASRA, ODPC, KRA, Financial Reporting Centre (GOAML). These can be acknowledged but every acknowledgement is itemised in the Cutover Go/No-Go report and visible to the Internal Auditor.
- KYC_MISSING_LARGE_BALANCE — member balance > KES 1,000,000 but no ID document on file. Triggers a CBK Risk-Based AML flag.
- GOAML_HIST_THRESHOLD — historical transaction >= KES 1,000,000 (cash) or USD 10,000 (FX). Should have been GOAML-reported in legacy; importer flags for retrospective filing.
- ODPC_CONSENT_MISSING — member created after 2019 (when the Data Protection Act came in) but no marketing-consent flag captured.
- KRA_PIN_MISSING_INTEREST_PAYER — member is paid savings interest > KES 24,000/year but has no KRA PIN; withholding tax cannot be remitted correctly.
- SASRA_PRODUCT_NOT_LICENSED — savings or loan product on the source is not in the SASRA product catalogue for this tenant's licence class.
Running validation
- Open the Run. Stage should be 'staged' or 'validated_with_warnings'.
- Click 'Run Validation'. A progress bar appears; for 25,000 members typical runtime is 2-4 minutes.
- When validation completes, the Run shows three numbers: Errors, Warnings, Clean rows.
- Open the Exception Report (button next to validation result). The report is a list view of mfi.migration.exception, filterable by severity, rule_code and resolved state.
- For each Error: click through to the staging row, fix it (either edit in place or fix the source file and re-upload), and re-run validation.
- For each Warning: review, then either fix or click 'Acknowledge' and enter a reason. The reason is required and limited to 240 characters.
- When Errors=0 and every Warning has resolved=True, the Run moves to ready_to_commit and the Commit button enables.
Note — Validation is idempotent — running it twice in a row produces the same exception list (with previously resolved warnings remaining resolved). Re-validation is recommended after any staging edit.
Worked scenarios
Scenario — Validation finds 47 loan schedule mismatches at Tujikomboe
| Character | Role |
|---|---|
| Jane Wambui | Migration Lead |
| Peter Otieno | Data Steward |
| Florence Achieng | Finance Reviewer |
| Mary Mutua | Credit Officer subject-matter expert |
Timeline
- Day 10, 10:00: Jane runs Validation on Run TJK-LOAN-002. (Result: 0 errors, 312 warnings, 2,809 clean. Runtime: 4m 12s.)
- Day 10, 10:20: Peter filters Exceptions by rule_code=SCHEDULE_SUM_MISMATCH. 47 hits. (Each row shows loan source_key, expected sum, actual sum, delta.)
- Day 10, 11:00: Peter and Mary inspect 10 sample loans. All 10 are loans where Loan Performer recorded a partial payment as a 'lump' txn without updating the schedule. (Pattern confirmed.)
- Day 10, 14:00: Jane asks Aisha to re-export schedules from Loan Performer with the 'Reconcile' option enabled. 47 schedules now sum correctly. (Run TJK-LOAN-002 cancelled; new Run TJK-LOAN-003 created.)
- Day 11, 09:00: Re-upload, re-stage, re-validate. (0 errors, 198 warnings (none about schedules), 2,923 clean.)
- Day 11, 10:30: Peter acknowledges 178 LOAN_OUTSTANDING_GT_PRINCIPAL warnings — these are loans with uncapitalised arrears interest. Reason: 'Carrying arrears interest forward per board resolution 2025-11-08.' (Acknowledgements logged.)
- Day 11, 11:00: Florence reviews the remaining 20 warnings — all KRA_PIN_MISSING_INTEREST_PAYER. Decision: capture PINs in iBANK during the first 30 days post go-live. (Acknowledgements logged with reason.)
- Day 11, 11:30: Run TJK-LOAN-003 reaches ready_to_commit. (Commit button enables.)
Outcome — Loans staging is validated. All 47 schedule mismatches resolved at source rather than papered over. 198 warnings acknowledged with documented reasons. Ready for Commit.
Reference
Validation rules catalogue (excerpt)
Full list of 73 rules across the six staging tables. Edit the catalogue at Settings -> Technical -> Migration -> Validation Rules.
| Code | Table | Severity | Owner role | Auto-acknowledgeable? |
|---|---|---|---|---|
| REQUIRED_MISSING | all | Error | Lead | No |
| TYPE_MISMATCH | all | Error | Lead | No |
| DUP_SOURCE_KEY | all | Error | Lead | No |
| FK_NOT_FOUND | savings, loan, schedule, txn | Error | Lead | No |
| AGE_OUTLIER | member | Warning | Steward | No |
| BALANCE_NEGATIVE_SAVINGS | savings | Warning | Steward | No |
| LOAN_OUTSTANDING_GT_PRINCIPAL | loan | Warning | Steward | No |
| SCHEDULE_SUM_MISMATCH | schedule | Warning | Steward | No |
| KYC_MISSING_LARGE_BALANCE | member | Reg Warning | Steward + Finance | No |
| GOAML_HIST_THRESHOLD | transaction | Reg Warning | Steward + Finance | No |
| ODPC_CONSENT_MISSING | member | Reg Warning | Steward | No |
| KRA_PIN_MISSING_INTEREST_PAYER | member, savings | Reg Warning | Steward + Finance | No |
| SASRA_PRODUCT_NOT_LICENSED | savings, loan | Error | Lead | No |
| PHONE_NOT_KENYAN | member | Warning | Steward | Yes (if explicitly toggled) |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Validation runtime exceeds 30 minutes. | Cross-table FK_NOT_FOUND check is doing a full scan because the staging table is missing an index. | Restart the BridgeERP worker (the indexes are auto-created on module install but a long-running worker may miss them); or run ANALYZE on the staging table. |
| Acknowledging a warning does not enable Commit. | At least one Error-severity exception remains, or the acknowledged warning's reason is shorter than the minimum 8 characters. | Re-run validation; expand the Errors filter; re-open the warning and write a fuller reason. |
| FK_NOT_FOUND on a savings row even though the member is in staging. | member_key in the savings row was uppercased while the member's source_key was lowercased (or vice-versa). | Edit Source mapping to normalise case (both .upper() or both .lower()), re-upload. |
| SCHEDULE_SUM_MISMATCH on every single loan with a tiny delta (1-3 KES). | Rounding difference between legacy and MFI Suite — legacy rounds per instalment, MFI Suite to the loan total. | Set the migration parameter mfi.migration.schedule_rounding_tolerance to 5 (KES). Re-validate. |
See also
Was this page helpful?