Apache Airflow is my favorite orchestration tool, and I’m very familiar with it. One feature I really like is sensors. They let us pause a task until a specific condition is met.
Some common use cases are:
Waiting for a file to land in object storage. We can even make the file path dynamic by using the current date in the naming.
Creating a custom sensor, for example, to check if a staging log record exists. If it does, the next task can run.
I find this feature powerful because it ensures tasks only run when conditions are met. If not, the sensor keeps pocking until the timeout or interval we configure.
In this blog, I will try to make sensor from the postgres logging table. Firstly, I create custom postgres sensor and put it into plugins directory.
And I create a simple dags like this, I call my custom postgres sensor and set into the task.
It will be always pocking when the table logs not met the criteria.
And now, I'm trying to insert the record for logging table, the sensor will be met the criteria.
