A twenty-minute network drop in the middle of a production run, and the line's readings vanish without a trace. That is the scenario that judges industrial IoT data ingestion on Azure: not nominal throughput, not the demo dashboard, the outage. Plant networks fail at the worst possible moment, and the readings you lose are exactly the ones quality will ask about the next morning.
We spend our days engineering industrial and manufacturing platforms on Azure, from the shop floor gateway to cloud storage. This article describes one solved pattern, the one that makes industrial IoT data ingestion into Azure survive real plant conditions: a queue between the shop floor and the cloud. It is written for the industrial IT manager who has to defend that choice in front of a plant director.
The promise: no reading lost, even with the network down, even when every machine reports end of shift within the same minute. The price is counted in engineering days, and we put numbers on it below on our public rate grid.
The failure mode first: the hole in your history
The naive setup holds for weeks. A shop floor service reads measurements from the PLCs and writes them straight into a database hosted in Azure. Then the link goes down: firewall maintenance, a telecom incident, a cable caught by a forklift. For twenty minutes every write fails. The service retries once, twice, gives up. Production, meanwhile, never stopped.
The result is a hole in the history, invisible at the time. You find it weeks later, during a customer claim or a drift analysis: the readings for the disputed lot do not exist. Machine data cannot be replayed. Tuesday's oven temperature at 14:12 is gone for good.
Why machines straight into a database is the default mistake
It is the default architecture because it is the fastest to build. One connection string, one table, one write per reading. In a demo, with one machine on an office network, nothing separates it from a good architecture.
It breaks for three reasons. The database becomes a single point of failure: when it is unreachable, production data is lost. Every machine knows the schema: the smallest structural change travels all the way down to the shop floor. And the end-of-shift burst, when every machine reports at once, hits the database with no shock absorber.
The pattern: a queue between the plant and the cloud
The diagram fits in one sentence. The machine talks to a gateway on the shop floor; the gateway writes to a queue in Azure; a consumer reads the queue at its own pace and writes to the database. Three links, and each link only knows the next one.
Each link absorbs a different failure. Network down: the gateway buffers readings on local disk and replays them on reconnection. Database unavailable: the queue holds the messages, the consumer resumes where it left off. Burst: the queue absorbs it, the consumer smooths it out. The database never sees the storm.
The decoupling has a practical consequence plant directors like: you can replace the database without touching the shop floor, and add a machine without touching the cloud.
Bursts, ordering, duplicates: the three questions that decide the design
The pattern is standard, its sizing is not. Three questions decide everything.
- What is the worst burst? Not the average rate: the reconnection after an hour-long outage, when the gateway replays its whole buffer on top of the end-of-shift peak. You size the queue and the consumer for that peak.
- Does ordering matter? For a measurement history, a timestamp taken at the source is enough and arrival order is irrelevant. For machine state calculations you need guaranteed per-machine ordering, which queue partitions handle.
- What happens to duplicates? Reliable delivery means at-least-once delivery: after an outage, some messages arrive twice. Writes must be idempotent, keyed on machine id plus timestamp. That is a design decision, not a last-minute patch.
The answers fit on one page and belong in the design before anyone writes code.
What the queue costs, and when simpler wins
The pattern is not free. It adds a few seconds of latency, which rules out real-time control through this path: control stays in the PLC. It also adds a component to operate: a queue has to be watched, with its depth, its consumer lag and its dead-letter queue for invalid messages.
A simpler path wins in two cases. If the machine keeps its own history and you can read it back later, a network drop is no longer data loss: periodic direct writes are enough. And if the readings feed a convenience display only, with no traceability or quality process behind them, the pattern's rigor is not justified yet.
Where industrial IoT data ingestion sits in an Azure estate you can run
A mid-size manufacturer will not operate an exotic streaming platform. The Azure version of the pattern fits in four managed services: a software gateway on an industrial PC in the plant, Azure IoT Hub or Event Hubs as the entry queue, Azure Functions as the consumer, a SQL database or historian storage at the end.
That footprint is manageable with the IT team a plant already has: no cluster to maintain, no specialist on-call rota. Our test: if the pattern demands a skill the plant does not have and will not hire, it is the wrong pattern.
When the pattern pays for itself in a fixed-scope project
Here is a first implementation priced on our public grid, 150 to 450 euros per day depending on the profile:
- Architecture and answering the three questions: 3 days at 450 euros = 1,350 euros
- Shop floor gateway with a disk buffer: 10 days at 300 euros = 3,000 euros
- Consumer and idempotent writes: 8 days at 300 euros = 2,400 euros
- Monitoring, alerts and dead-letter handling: 5 days at 400 euros = 2,000 euros
- Failure drills, simulated outages and go-live: 4 days at 400 euros = 1,600 euros
Total: 30 days and 10,350 euros. The direct path costs roughly ten days less. The gap pays for itself at the first serious outage once your readings feed traceability, quality or billing: one hole in the history of a sensitive lot costs more in investigation time than ten days of engineering.
The scope holds in a fixed-price project because the pattern is known: the unknowns concentrate in the machine protocols, not in the architecture. For a multi-line program we switch to a pod, a stable team at 15,000 to 40,000 euros per month. We run a hybrid model: steering, quality and contracts in Belgium, engineering in Rabat. More than 17 projects shipped for more than 10 clients. The earliest start is two weeks after your request.
Run your own plant through this pattern on jadev-corp.com/quote: describe your machines and your flows, get an instant estimate on the same rate grid, and an engineer reviews every formal quotation within one business day.