The pipeline I am describing is the flow of information that starts where work is performed, informs a decision-maker, who then adjusts how the work is done, which in turn updates the decision-maker on the result, creating a continuous feedback loop. The intent is to improve performance. Drawing from my background in Lean Manufacturing, I see two critical insights.

The first insight is the historical reduction of the data pipeline latency. Early factories relied on time and motion studies to create data points for improving the efficiency of workers and machines, essentially creating a very manual and arduous data pipeline from work to the manager. This pipeline later matured within the automotive industry with the adoption of methods like Plan-Do-Check-Act, Lean, Six Sigma, and Theory of Constraints, lowering the information latency with the social and technical systems of their time. After the digital age erupted, these methods matured the data pipeline further with Agile and DevOps, taking direct inspiration from the manufacturing industry, focusing on tasks and code rather than widgets, creating formal data pipelines closer to what we see today. Ironically, the concepts are reemerging in manufacturing, with Model-Based Systems Engineering incorporating IoT, additive manufacturing, process mining, and digital twins. What we see in retrospect is a vast improvement in feedback latency between work and decision-maker, where improvements in a traditional six sigma project might take months, agile can implement in weekly sprints, DevOps within minutes, and AI is now opening the possibility for near-zero latency.

The second insight is the outlier of the Toyota Production System (TPS) from the timeline I described. Toyota developed an analog data pipeline that rivals modern standards and established itself as a global competitor, despite developing in an economically devastated post-World War II Japan. How was this accomplished? Agile's concept of "Low-Tech, High-Touch" was inspired by this era. Kanban tightened inventories by limiting work in progress with simple visual signals. The Andon Cord allowed workers to stop the assembly line, prompting a quality team to swarm the problem. Kaizen encouraged the assembler to be the decision maker for small incremental improvements. Genchi Genbutsu pushed "management by walking around" rather than giving orders from an office. These feedback loops of improvement were entirely human-centered. The magic of TPS isn't in the complexity of its tools but in the clarity of its signals.

Now consider the data pipeline of your work. What are you measuring? How are you using it to improve the work? The data pipeline needs to inform the decision-maker and enable action. Whether you're using a stopwatch to time processes, sensors to track throughput, or a neural network to crawl timestamps, if the tool isn't amplifying the signal, it's automating the noise.