WRITING
Notes from production
Things I learned the slow way, mostly about wiring systems together that were never designed to talk to each other. ERP integration, messaging infrastructure, and the unglamorous work of making either one reliable.
Latency in 100ms steps
I spent a day tuning Elasticsearch queries. The slowdown was CPU throttling from our VPS host, triggered by a Logstash container that had crash-looped 41,687 times.
Too slow is too big
My fallback for large Tally exports waited for a "too big" error. The biggest company never sent one. It timed out at Cloudflare's 120 seconds instead, and the fallback never ran.
The floor was the call
An LLM classifier took three seconds to answer even "confirm". Shrinking the output barely helped, because the model's fixed per-call cost was the floor. A faster model was right there, and I still didn't take it.
Five agents, three human approval gates
A multi-agent LLM system that spends real money on ads: every ad created paused, three human approval gates, and every model call logged.
Ten seconds to two
A WhatsApp chatbot took ten seconds to reply. Twelve serial reads, a blocking typing indicator and an LLM call on the critical path. Getting it to two.
The index Postgres quietly stopped using
A refactor made catalogue search 54% slower at p90. The partial index no longer matched the query, so Postgres quietly skipped it. 281 ms to 29 ms.
Your HTTP timeout is probably not a timeout
axios' timeout is a socket-idle timeout, not a deadline. A trickling response never goes idle, so it never fires. Plus a 26 MB parse that outran its own timer.
One WhatsApp number, many stores
One WhatsApp Cloud API number serving many stores: resolving the tenant from a webhook that doesn't name one, and caching it without guessing wrong.
Building a two-way Tally ERP sync that survives retries
Tally has no REST API, answers HTTP 200 on failure, and its documented batching is a no-op. What two generations of sync taught me.