Integration sounds like a single task. It is really four decisions, and getting any of them wrong produces a system that works in the demo and drops records in production.
Push or pull
Webhooks (push). The source system calls your endpoint the moment something happens. Near-instant, no wasted calls. The catch: you need a reachable, always-available endpoint, and you must handle the case where you were down when it fired.
Polling (pull). You ask every few minutes what changed. Simpler, works behind a firewall, no public endpoint. The catch: latency, and you burn API quota asking a question whose answer is usually "nothing".
Use webhooks where the platform supports them and speed matters — a new lead should not wait five minutes. Use polling for reconciliation, and for platforms with no webhook support.
The robust pattern is both: webhooks for immediacy, a nightly poll to catch anything missed.
Idempotency, or the duplicate lead problem
Webhooks get delivered more than once. This is normal, not a bug — the sender retries when it does not get a clean acknowledgement, and sometimes your response was lost even though you processed it.
If you insert a row for every delivery, one enquiry becomes three, an agent calls the same person twice, and trust in the system evaporates.
The fix is boring and essential: give every incoming event a stable identifier, record the ones you have processed, and ignore repeats. Where the source supplies no id, derive one from the content — the submission timestamp plus the email address is usually enough.
Retries and failure
The receiving system will be down sometimes. Yours will too.
- Acknowledge fast, process after. Accept the webhook, write it to a queue, return 200 immediately. Do the slow work separately. Platforms disable endpoints that time out.
- Retry with backoff. Not every second — after 1, 5, 25 minutes. A failing endpoint hammered by retries stays failing.
- Give up loudly. After a few attempts, park it in a dead-letter list a human can look at. Silent failure is worse than a visible error.
Field mapping is where the time goes
Everyone underestimates this. Two systems never agree on shape.
One has a single name, the other wants first_name and last_name. One's status values are new/contacted/won, the other's are numeric ids. Phone numbers arrive in five formats and one side demands E.164.
Write the mapping down before writing any code, including what happens when a required field on the destination is empty on the source. That document is the actual specification; the code is a translation of it.
What to log
When a lead goes missing — and one will — you need to answer three questions: did it arrive, did we transform it correctly, did the destination accept it.
So log the raw payload as received, the transformed version, and the response. Not forever, but long enough to investigate. Almost every "the integration is broken" report we get is answered in minutes by those three records, and is usually a mapping problem rather than a connectivity one.
The honest scoping question
Before building any of it, ask whether the integration is worth it. If a lead arrives once a week and someone already checks that inbox, a webhook pipeline with retry logic is engineering for its own sake.
Integrate when volume makes manual handling unreliable, when speed changes the outcome, or when two systems disagreeing causes real problems. Otherwise, an export button is a perfectly respectable answer.
