Hi S***n,
Great questions,here’s a bit more on how we keep tests and dashboards in lock-step as things evolve:
1. Syncing Great Expectations across squads
• Centralized test library: all GE specs (schema checks, business‐rule validations) live in a shared Git repo alongside our ELT code. Tests are organized by data domain (e.g. Payments, Settlements) and parameterized off our canonical schema definitions.
• Versioned schemas & tests: whenever a squad proposes a schema or rule change, they create a Git branch with updated DDL, corresponding GE tests and a Confluence spec. A CI pipeline runs those tests in a “sandbox” environment, flags any failures, and only merges once the Data Governance Council signs off.
• Cross-squad coordination: we hold a biweekly “Data Quality Forum” where squad leads review planned schema updates, align test coverage and highlight any downstream implications,so nobody drifts out of sync.
2. Managing Athena view versioning & QuickSight promotions
• Git-based view definitions: every Athena view lives in version control, with semantic version tags. When you update a view, you branch, update DDL, run regression tests against a shadow‐table pipeline, then merge once approved.
• CI/CD for QuickSight assets: we store QuickSight dataset and dashboard definitions in code (via AWS CloudFormation/CLI). When a view is promoted from dev → test → prod, our pipeline automatically updates the underlying dataset references (using stable view identifiers), runs integration smoke-tests, and then deploys the new dashboards,ensuring end users always see metrics aligned to the latest schema.
• Environment isolation & governance: dev/test/prod each live in separate AWS accounts/QuickSight namespaces, and promotions require a Data Council sign-off step in Jira Service Desk.
If you’d like more granularity or a live demo of these workflows, I’d be happy to loop in our Head of Data Engineering on Thursday’s call. Let me know, and I’ll coordinate calendars.
Cheers,
J***e | HR Recruiter, StrategyBrain