If you trigger a DAG in the Web UI, the Webserver will update the DAG in the database accordingly. For instance, the Webserver reads the current state of the DAG execution from the database and displays it in the Web UI. Instead, they all read and modify the state that is stored in the shared database. The following diagram depicts the Aiflow architecture on OpenShift:Īs shown in the architecture diagram above, none of the Airflow components communicate directly with each other. After the task execution is complete, the Worker pod is deleted. They are created by the Kubernetes Executor and their sole purpose is to execute a single DAG task. On the other hand, Airflow Workers - the last of the three main components - run as ephemeral pods. Both Airflow Webserver and Scheduler are long-running services. The second component - the Airflow Scheduler - orchestrates the execution of DAGs by starting the DAG tasks at the right time and in the right order. In addition to the Web UI, the Webserver also provides an experimental REST API that allows controlling Airflow programatically as opposed to through the Web UI. It allows users to visualize their DAGs (Directed Acyclic Graph) and control the execution of their DAGs. The Webserver provides the Web UI which is the Airflow’s main user interface. The three main components of Apache Airflow are the Webserver, Scheduler, and Workers. This article focuses on the latest Apache Airflow version 1.10.12. We are going to discuss the function of the individual Airflow components and how they can be deployed to OpenShift. This blog will walk you through the Apache Airflow architecture on OpenShift.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |