How tenant data becomes search-ready
A high-level view of the controlled data path that prepares each tenant's catalog before it is indexed for Querix search.
01
What the pipeline owns
The pipeline is deliberately separate from online search. It reads tenant source data, applies tenant-specific normalization, produces retrieval text and structured metadata, validates the output, and can publish a complete validated dataset to an isolated serving destination. Embeddings are generated later by the search ingestion service.
- 01
Read-only source database
- 02
Source refresh and local snapshots
- 03
Tenant adapter and normalization
- 04
Retrieval-content enrichment
- 05
Validation and Parquet artifact
- 06
Optional atomic publish
Tenant data boundary
Source access is read-only. Each tenant's source, processing artifacts, and serving destination remain isolated, and validated output is handed to search indexing only after the data-quality checks pass.
02
The staged transformation model
A tenant profile selects the source/destination contracts and adapter. The pipeline builds one traceable artifact through ordered stages, so a failure can be resumed at the affected stage instead of repeating safe completed work.
| Field or role | Type or store | Contract |
|---|---|---|
| Source sync | Snapshot | Reads source tables, compares changes, and maintains a current local input snapshot. |
| Canonical catalog | Normalize | Applies tenant adapter rules and produces stable IDs and core catalog records. |
| Category, location, attributes | Enrich | Adds structured metadata used by filters and search interpretation. |
| Embedding-ready | Compose | Builds normalized retrieval content and filter metadata without generating embeddings. |
| Search-ready | Validate | Produces the final Parquet dataset consumed by downstream search ingestion. |
| Publish | Promote | Loads a staging table and atomically promotes the validated destination dataset. |
03
Freshness without exposing internal operations
The first successful run creates the baseline required for later change detection. It reads the configured source, backs up the previous local snapshot, runs every transformation stage, and validates the final artifact. It does not publish to a tenant destination unless publishing is explicitly requested.
04
Incremental change handling
Subsequent runs compare source snapshots and rebuild only new, changed, dependent, or deleted records. No-change runs skip expensive transforms. A missing baseline triggers a safe full build; shared category, location, or attribute reference changes also trigger a full rebuild because they can affect many records at once.
- New and changed source records are rebuilt.
- Deleted records are removed from the next search-ready artifact.
- Dependent-record changes rebuild the affected catalog entries.
- The full merged final artifact is retained for atomic destination replacement.
05
Validation before search
Validation is a hard gate: a failed validation cannot publish. Successful output is stored as a Parquet artifact, which is compact and suitable for production servers. When publish is requested, the destination write uses a staging table and atomic promotion rather than exposing a partially-loaded serving table.
Validation report
Confirms final output correctness before any destination write.
Search-ready artifact
The durable Parquet handoff from ETL to downstream retrieval indexing.
Atomic promotion
Avoids presenting a partially loaded destination table to search consumers.
- Stable catalog IDs are present and unique, with no cross-tenant rows.
- Retrieval text and keyword content are present for every searchable record.
- Category, location, and attribute mappings meet the tenant's accepted quality baseline.
- Unexpected row-count drops, join regressions, or unusually large source changes stop promotion for review.
06
Reliable lifecycle management
Refreshes are guarded so one tenant's processing cannot overlap with itself. The operating workflow can resume from the last safe stage rather than repeating successful work, keeping the data lifecycle reliable without exposing operational controls to client users.
Validation failures are isolated from source data and completed transformation stages. The pipeline can safely recover from the affected validation point without unnecessarily re-reading tenant data or recomputing finished enrichment.
07
What this means for a tenant
The data lifecycle produces traceable quality and change records for the Querix operations team. Tenant systems retain ownership of their canonical data, while Querix prepares the validated search-ready representation required for accurate discovery.
- Tenant source data stays isolated from other tenants.
- Normalization creates consistent catalog and filter metadata.
- Validation prevents malformed or incomplete data from reaching search.
- Only validated search-ready data moves into the retrieval indexing path.
- Canonical tenant records remain the source of truth for results returned to users.