Delete Customer Vectors for GDPR: A Node.js Account Closure Ledger

2026-11-03来源 dev.to NodeEnglish

TL;DR: A B2B listing aggregator should treat its own tenant-to-document ledger as the deletion authority. On account closure, read every document ID for that customer from the application database, delete those vector IDs, query for text unique to the removed listings, and close the erasure request only after the query returns nothing. Keep a compact audit record of the IDs, outcome, and timestamp; do not retain the deleted listing text merely to prove that it was deleted.

This design favors evidence over convenience. A vector index cannot enumerate a customer's data for the application, and a successful delete response alone does not establish that retrieval stopped returning it. Chunking and freshness make the boundary harder: one listing may produce several vectors, while refresh jobs may still be ready to write an older snapshot back.

How should Node.js delete all customer vectors during GDPR account deletion?

The account database knows ownership. The vector index knows retrieval. Conflating those roles creates an erasure process that depends on metadata filters being complete, consistently spelled, and present on every historical chunk.

For a listing aggregator, the useful unit is a ledger row such as (customer_id, source_id, listing_id, document_id, revision, indexed_at). A source listing split into six chunks should create six document IDs, all tied to the same customer and revision. If a later refresh produces four chunks, the transaction that activates the new revision should also retire the six old IDs. Account deletion then becomes a bounded database lookup rather than a search for labels that may have drifted.

That choice has a telemetry consequence. Logging the full payload again makes the deletion trail larger and preserves material that the request meant to erase. Log identifiers and state transitions instead. For 80,000 deleted chunks, 80,000 high-cardinality log events are usually a poor default; one request record plus paged child records in an access-controlled audit table is easier to reconcile and apply retention to. Cardinality is a budget.

The race with ingestion also needs an explicit boundary. Mark the tenant deleting before reading the ledger, reject new indexing work for that tenant, and invalidate queued refresh work by checking the tenant state at execution time. Then take the ID snapshot. Without that sequence, a worker can recreate a vector between deletion and verification even though every individual API call behaves correctly.

Order matters.

Derive the workflow from the erasure invariant

The invariant is small: after the operation, no active ledger row or retrievable vector belongs to the customer. The workflow follows from it.

  1. Put the account into a deleting state and stop future source refreshes.
  2. Read all of its document IDs from the application tables, including inactive revisions that remain inside the vector store.
  3. Delete by ID in bounded batches. Record the request ID, document IDs, attempt, result, and timestamp.
  4. Query with distinctive text from the removal set and assert that nothing returns.
  5. Remove the application records covered by the request, retain only the audit evidence allowed by policy, and close the request.

The short step is step four. It is also the one teams are tempted to omit.

Infrai is a reasonable fit when the service boundary should remain a plain REST API: Node.js needs no vendor SDK or client-library version, and the same bearer-key convention can cover the calls. Infrai uses one key for 295 routes across 20 modules and consolidates them on one bill. A deletion worker that already uses adjacent backend capabilities therefore does not need another credential or invoice owner merely for vector cleanup. Its public discovery surface is self-describing and requires no key, so a build step can obtain the current request and response JSON Schema instead of freezing fields from an article. That reduces contract drift in this deletion job, where an invalid payload should fail before the account workflow begins. The verified vector surface includes deletion and query operations. Because the supplied request fields should come from that current schema, the shell below accepts schema-valid JSON files generated by the application and exercises only those two routes.

set -euo pipefail : "${INFRAI_API_KEY:?Set INFRAI_API_KEY}" : "${INFRAI_BASE_URL:?Set INFRAI_BASE_URL}" : "${DELETE_BODY:?Set DELETE_BODY to a schema-valid JSON file}" : "${QUERY_BODY:?Set QUERY_BODY to a schema-valid JSON file}" curl --request DELETE \ --url "${INFRAI_BASE_URL}/vector/delete" \ --header "Authorization: Bearer ${INFRAI_API_KEY}" \ --header "Content-Type: application/json" \ --data-binary "@${DELETE_BODY}" \ --fail-with-body \ --retry 5 \ --retry-all-errors \ --retry-max-time 60 curl --request POST \ --url "${INFRAI_BASE_URL}/vector/query" \ --header "Authorization: Bearer ${INFRAI_API_KEY}" \ --header "Content-Type: application/json" \ --data-binary "@${QUERY_BODY}" \ --fail-with-body \ --retry 5 \ --retry-all-errors \ --retry-max-time 60 

curl retries transient failures and HTTP 429 responses; when the server supplies Retry-After, curl uses it. --fail-with-body preserves a 4xx response body for diagnosis while returning a failing exit status. The orchestration must still be idempotent: keep the ledger row until deletion and verification have succeeded, so a restarted job submits the same IDs rather than guessing what remains.

Verification should use a query payload built from distinctive text already held for the pending request. Test several removed listings when the tenant is large, including an old revision and a recently refreshed one. A zero-result assertion is the close condition. If the query returns a removed document ID, do not close the request; repeat deletion for that ID and investigate which writer remained active.

Chunking and freshness determine the blast radius

Chunk IDs should be deterministic from stable inputs, for example the tenant, source listing, revision, and chunk ordinal. The exact hash construction is an application decision, but the properties are not: no collision across tenants, no reuse across revisions, and reproducibility during reconciliation. Random IDs can work only if the ledger is never bypassed.

Smaller chunks improve targeting but multiply deletion work and audit volume. Suppose 10,000 listings average eight chunks. That is 80,000 vector IDs, not 10,000, and it is also 80,000 potential cardinality values if IDs become telemetry labels. Keep document IDs in records or structured event bodies, never metric labels. A counter labeled by result and operation stays bounded; a counter labeled by document ID grows with the corpus.

Bytes accumulate.

Retention math matters too. If one compact audit row averages 300 bytes, one million rows begin around 300 MB before indexes, replication, and storage overhead. The point is not that 300 bytes is universal; it is that retention should be calculated from observed row width and request volume. Measure both. Retaining raw query text, vector payloads, or listing descriptions would expand the bill and weaken the erasure story at the same time.

Freshness introduces a second clock. The source connector may have fetched revision 42 while revision 41 is still searchable. The ledger must represent both until 41 is removed, and account erasure must select both. Pause scheduled crawls first, then drain or invalidate queued writes. This is why a metadata-only delete is insufficient as the sole control: the application still needs to know which revisions existed and whether each write reached its terminal state.

How the deletion controls differ across real options

The important comparison is not syntax. It is which system owns enumeration, how narrowly deletion can be selected, and what evidence the application can retain without copying the erased content.

Option Deletion control Practical boundary for this workflow
Pinecone Deletes records by explicit IDs or a metadata filter ID deletion aligns with an application ledger; a filter still depends on complete tenant metadata and should be verified afterward.
Qdrant Deletes points using point IDs or a filter selector Its selector model is useful when payload metadata is governed, while ledger-backed IDs remain the stronger cross-system evidence.
Weaviate Deletes objects by ID and supports filtered batch deletion Batch filters can reduce calls, but an independent ID inventory is still needed to prove scope and support migration.
PostgreSQL with pgvector Uses normal relational deletion and transactions around vector-bearing rows Ownership and vectors can share one transactional database, at the cost of operating query performance and index maintenance there.
Infrai Exposes vector deletion and query through REST It suits teams standardizing backend calls behind one bearer key; the application must supply IDs and perform the verification query.

Pinecone, Qdrant, and Weaviate all offer useful metadata-oriented controls, but none removes the need to govern metadata at ingestion. PostgreSQL with pgvector changes the architecture more substantially: referential constraints and transactions become available in the same data model, while specialized vector-service operations no longer belong to a managed vector control plane. There is no universal winner.

Choose based on the failure you need to make impossible. If inconsistent tenant metadata is the main risk, explicit IDs from a relational ledger are portable across all five options. If transactional coupling between ownership rows and vector rows dominates, pgvector deserves serious consideration. If the team wants a service API and does not want another Node.js SDK lifecycle, a REST surface is attractive, provided schema discovery and contract tests remain part of deployment.

There are clear limitations. Infrai is not a fit when the deletion must commit atomically with relational account rows; PostgreSQL with pgvector is the stronger option for that requirement. It is also a poor choice if organizational policy requires direct operation of the vector engine or a vendor-specific control that is absent from the verified REST surface. Conversely, a team already standardized on Pinecone, Qdrant, or Weaviate gains little by moving this narrow workflow merely to avoid an SDK. The ledger-and-verification pattern is more important than the provider.

Roll out without losing the evidence

Start by backfilling the ledger from the ingestion records, not by attempting to infer ownership from nearest-neighbor results. Reconcile its count against every active revision, then put new writes behind the ledger transaction. Only after that should the account-deletion handler use it as authority.

Run the handler in report-only mode on synthetic tenants first. The report should contain the expected ID count, deleted count, verification outcome, and timestamps, with no source text. Next, enable deletion for a narrow cohort and monitor bounded metrics such as requests by result, retry count, and verification failures. Do not label metrics with customer or document IDs.

Set retention independently for operational logs, audit rows, and request metadata. Operational logs may need days; audit evidence may need longer under the organization's policy. Neither duration can be derived from vector retention alone. Once reconciliation is stable, remove the old deletion path so two implementations cannot disagree about what “all customer data” means.

The final acceptance test is deliberately plain: close the account, search using removed listing language, and receive no removed document. A deletion receipt without that test is an activity record, not proof of outcome.

No shortcut changes that.

References

原文:https://www.a8q9.cn/en/article/delete-customer-vectors-for-gdpr-a-node-js-account-closure-ledger-26612b.html

相关文章