An external verification API should be treated as a fallible dependency. Networks stall, credentials expire, quotas fill, schemas evolve, and receiving mail systems return ambiguity. Reliability comes from containing those conditions so they do not become false address decisions, duplicate work, or an unavailable signup flow.
Set timeout budgets
Start from the total user or job deadline, then allocate only part of it to verification. Use separate connection and response timeouts where the client permits. A server-side timeout must be shorter than an upstream proxy or browser deadline so the application can choose a controlled response.
Batch workers can allow longer operations than interactive forms, but every request still needs a limit. An unbounded socket can exhaust worker capacity.
Identify retryable failures
Connection resets, selected timeouts, explicit service-unavailable responses, and documented rate-limit responses may be retryable. A retry is justified only when the operation is safe to repeat and time remains in the workflow budget.
Record a broad reason and attempt count. Never turn exhausted retries into an invalid email result.
Identify non-retryable failures
Malformed requests, unsupported input, and most authentication or permission failures require correction, not repeated traffic. A response that validly classifies the address is an outcome even if the business dislikes it. Keep transport failure, configuration failure, and verification result as different types.
Alert on authentication failures quickly because they can affect every request. Redact credentials and addresses from alerts.
Respect rate limits
Limit concurrency locally and follow documented server guidance. If a response supplies a safe retry time, honour it within a bounded queue. Coordinate workers so each one does not independently create a request storm.
Track rate-limit events and queue age. A growing queue may require reduced intake or a degraded mode rather than more retries.
Use exponential backoff
Backoff spaces attempts so a brief incident can recover. A common policy increases delay after each failure up to a maximum. The exact numbers should match the workflow deadline and provider guidance, not a copied formula.
Interactive requests often permit no retry or one quick retry. Background jobs can wait longer. Always cap attempts and total age.
Add jitter
When many clients fail at once, identical backoff schedules wake them together. Jitter introduces a random component so recovery traffic spreads over time. Use a well-understood library implementation and test the range.
Randomness should affect scheduling, not the final business decision. Log the chosen next-attempt time for diagnosis.
Cache with an expiry
Caching can reduce cost and latency when the same address is checked repeatedly. Key the cache using a protected normalised representation and include policy or provider version where changes matter. Encrypt or otherwise protect stored personal data.
Use different lifetimes for stable and transient results. Do not cache a timeout as invalid. Any result can age, so define expiry and explicit invalidation triggers such as a confirmed address change.
Make operations idempotent
A repeated job should not create duplicate contact updates, charges in your own ledger, or repeated downstream events. Assign an internal request or job ID and store completion against it. For batch imports, use the source row ID plus import version.
Do not assume a provider supports an idempotency header unless its current documentation says so. Enforce safety in your boundary even when the external service lacks that feature.
Preserve unknown results
Unknown is an honest outcome when evidence is ambiguous. Map it to review, confirmation, or a bounded retry according to context. It must never be collapsed into invalid for convenience.
Track reasons within the unknown category. Provider opacity and your own timeout need different operational responses.
Design useful logging
Log request IDs, internal workflow, provider, duration, attempt, outcome category, cache status, and broad error class. Avoid full email addresses, request bodies, credentials, and raw responses in general logs. Where correlation is needed, use a controlled token with an appropriate threat model.
Set retention and access rules. Debug logging enabled during an incident can quietly become a shadow contact database.
Choose metrics
Measure request count, latency percentiles, timeout rate, provider error rate, rate-limit events, cache hits, unknown distribution, retry attempts, queue depth, and final decision counts. Segment by workflow because a batch job can hide an interactive outage in aggregate data.
Changes in valid or invalid proportions may reflect input mix. Treat them as signals for investigation, not proof of service quality.
Alert on user and queue impact
Alerts should reflect sustained failure, latency, queue age, or user impact. One isolated timeout may need only a metric. Authentication failures, schema parse failures, and a circuit breaker opening deserve prompt attention.
Include a runbook link, affected workflow, safe degraded mode, and owner. Never put private addresses in paging messages.
Use circuit breakers
A circuit breaker stops repeated calls when failure crosses a threshold, allows a recovery period, and then tests limited traffic. It protects worker capacity and the dependency from a retry storm. Choose thresholds using real traffic and avoid one global breaker that lets a batch incident disable every workflow.
The open state needs an explicit application behaviour such as accept-and-review, queue, or controlled pause.
Choose fail-open or fail-closed per operation
Fail open accepts the event when verification is unavailable and marks it for later handling. Fail closed stops or pauses the event. Ordinary signup may favour fail open with confirmation. A high-risk action can justify a closed or queued state when another proof is unavailable.
Do not let a programmer's default exception path decide policy. Product, security, privacy, and operations should agree on the consequence.
Safe pseudocode
function assess_email(address, context):
local = validate_syntax(address)
if local is clearly_invalid:
return decision("correct_input", source="local")
cached = result_cache.get(address, context.policy_version)
if cached is fresh:
return policy.map(cached, context)
if circuit_breaker.is_open(context.workflow):
return degraded_decision(context, reason="service_unavailable")
try:
response = verifier.check(address, timeout=context.timeout_budget)
result = validate_and_map_response(response)
result_cache.store_with_expiry(address, result)
return policy.map(result, context)
except RetryableFailure as error:
retry_queue.enqueue_once(context.request_id, address, error)
return degraded_decision(context, reason="retry_scheduled")
except ConfigurationFailure as error:
alert_without_personal_data(error, context.workflow)
return degraded_decision(context, reason="configuration_error")
The pseudocode deliberately keeps policy outside the vendor client. It does not invent an endpoint, field, or authentication scheme. Production code also needs input limits, secret management, cancellation, tracing, and tests for every exception branch.
Release checklist
- Timeouts are lower than the caller's deadline.
- Retries are bounded, jittered, and safe to repeat.
- Rate limits enter a queue rather than an invalid status.
- Cache entries expire and transient failures are not negative-cached.
- Logs and alerts exclude credentials and full addresses.
- Metrics separate workflows and outcome from integration failure.
- The circuit breaker and degraded mode are tested.
- Schema changes fail visibly without blocking every user.
- Queue reconciliation proves no work was lost or duplicated.
Mailthentic publishes an API product overview for teams evaluating that dependency. The reliability patterns here apply to any provider and should remain in your system: explicit budgets, stable mappings, observable failures, and decisions that preserve uncertainty.
Test failure behaviour
Build deterministic tests with a fake provider boundary. Simulate slow connections, timeouts after a response begins, rate limits with and without retry guidance, authentication failure, malformed JSON, missing fields, duplicate webhook events, and a valid unknown result. Assert the internal action and ensure no branch converts infrastructure trouble into an invalid address.
Load tests should include bursts and recovery, not only steady successful traffic. Confirm concurrency limits, connection-pool size, queue age, and breaker isolation between interactive and batch work.
Reconcile background work
A retry queue needs a periodic reconciler that finds stuck, expired, and duplicate items. Compare accepted jobs with completed actions using stable IDs. Move exhausted work to a visible final state with an owner rather than leaving it hidden in a dead-letter store.
Define what happens when the original customer record is deleted or its address changes while a retry waits. The worker should discard obsolete work and respect current retention and suppression state.
Plan provider changes
Pin or validate the response contract, subscribe to change notices, and test upgrades in a non-production environment. Keep raw provider fields at the adapter boundary so application code does not depend on an undocumented detail.
Feature flags can direct limited traffic to a new adapter, but results from different providers must remain clearly identifiable. Monitor decision distribution and error rates during the change, then keep a rollback window.
Comments (0)
No comments yet. Be the first to share your thoughts!
Leave a Comment
Your email address will not be published. Required fields are marked *