Healthcare Appointment Retrieval Deduplication Explained in Python: 5 Feeds (Trade-offs)
Healthcare Appointment Retrieval Deduplication Explained in Python: 5 Feeds (Trade-offs)
Short answer: deduplicate appointment evidence before ranking by assigning each source a canonical identity, then keep a separate ledger of the records merged into that identity. For a healthcare appointment assistant, this reduces repeated slots and contradictory citations without hiding which scheduling system supplied each claim.
The useful unit is not a text chunk. It is a source-backed fact: clinic, service, location, provider, time window, and effective date. A cancellation feed and a public booking page may describe the same slot with different wording. If both survive retrieval, the language model sees confidence where there is only duplication.
The failure starts before the vector index
I initially treated duplicate detection as a similarity-search problem. That was a mistake. Embeddings can tell me that two documents talk about “Tuesday dermatology appointments,” but they cannot safely decide that the records refer to the same clinic, timezone, or booking authority. A false merge can erase a valid slot; a missed merge can make the assistant cite two answers.
Normalize fields at ingestion. Parse timestamps into UTC while retaining the clinic timezone, normalize phone numbers and URLs, and map provider identifiers through a controlled table. Keep the original payload and an ingest timestamp. Those details are boring until an auditor asks why a slot disappeared. In a real appointment stream, the same provider can arrive as “Dr. Lee,” “LEE, M,” or an internal UUID; collapsing those labels without a maintained crosswalk is a data-governance decision, not a clever string trick. The crosswalk needs an owner, a version, and a review path when a clinic changes its naming rules.
For five hypothetical feeds, I use a key made from stable identifiers rather than generated prose:
from dataclasses import dataclass from hashlib import sha256 import json @dataclass(frozen=True) class AppointmentRecord: clinic_id: str service_code: str provider_id: str starts_at_utc: str source_system: str source_record_id: str payload: dict def source_key(record: AppointmentRecord) -> str: stable = { "clinic": record.clinic_id.strip().lower(), "service": record.service_code.strip().upper(), "provider": record.provider_id.strip().lower(), "starts_at": record.starts_at_utc, } encoded = json.dumps(stable, sort_keys=True, separators=(",", ":")).encode() return sha256(encoded).hexdigest() The hash is an implementation detail. The normalized fields are the contract. If a source cannot provide a stable provider or service identifier, mark the key as provisional and require a policy decision before merging it with a different authority.
Tiny detail, big consequence.
How should a healthcare appointment assistant resolve duplicate sources?
Use a two-stage pipeline. First, exact keys collapse records that represent the same appointment. Second, a source policy chooses the winning value for each field and records every losing record in a citation ledger. Retrieval then indexes one canonical record plus a compact list of provenance IDs.
A ledger entry should answer four questions: which records were merged, which source was authoritative for availability, when the decision was made, and what changed between versions. That makes the answer explainable without stuffing five copies of the same paragraph into the prompt.
Do not silently merge disagreements. If two systems report different times, retain both candidates, lower confidence, and route the turn to a clarification flow. Appointment booking is a consequential action; a fluent guess is not resolution.
Ranking quality versus response latency
Deduplication changes the retrieval distribution, so measure it as part of the search experiment. I track recall of the correct appointment, duplicate-rate in the top five results, time to first token, and citation agreement. The right threshold depends on the clinic data; your mileage may vary until you test representative cancellations, reschedules, and daylight-saving transitions.
A small evaluation set should include adversarial pairs: same provider and time but different locations, same slot with stale status, and two records that share a title but belong to different departments. Store expected identity groups as labels. Run the set on every parser or key change, and inspect failures by category rather than relying on one aggregate score.
Latency has a visible shape here. Exact-key grouping is cheap and bounded by the number of records. Fuzzy matching across the entire corpus is slower and harder to audit, so reserve it for a narrow candidate window after filtering by clinic and date. Cache normalized records, not model answers; cached answers can outlive a cancellation.
The catch: when source deduplication is the wrong choice
A single canonical record is not suitable when sources have independent legal or operational authority, or when the user explicitly asks for alternatives across locations. In those cases, preserve separate evidence and label the authority in the prompt. Stick with a broader multi-result retrieval strategy when freshness is more important than compact context.
There is also a boundary around identity resolution. If the input lacks a trustworthy identifier, the safe result is “possible duplicate,” not an automatic merge. Human review, a source-maintained crosswalk, or a later booking confirmation can resolve that uncertainty.
Keep the deduplication module independent from the embedding provider and the chat model. Its inputs and outputs should be plain JSON so a notebook fixture can become a batch job or an API worker without rewriting the policy. Log key versions, source IDs, and decision reasons; redact patient data from traces and enforce retention limits appropriate to your jurisdiction. Before launch, rehearse a stale-feed event, a provider rename, a timezone migration, and a partial replay. The assistant should state what it knows, cite the canonical source, and ask a focused question when identities remain ambiguous. That behavior is more valuable than squeezing another decimal point from a benchmark.