The majority of the helm release config is identical, although re-factored slightly to conform to the chart specific templates. In my opinion this is one of the down-sides of using helm to deploy applications in a smaller environment, however discussing the pros and cons of deploying to kubernetes uisng helm is not within the scope of this post. The configuration for the migration was very similar however we opted to use the official helm chart for the deployment rather than the community chart, which meant re-factoring many of the parameters and values to conform to the official helm chart specs. apiVersion: /v1alpha3Īll is working as expected, and when navigating in a web browser to you are routed to the Airflow web gui and are prompted to login using Google SSO identity provider. Our web base URL is configured with the HTTPS protocol: web:Īnd we have an Istio virtual service for ingress using our configured Istio exgternal ingress gateway. name: AIRFLOW_GOOGLE_OAUTH_CALLBACK_ROUTE Note: The AIRFLOW_GOOGLE_CLIENT_ID, and AIRFLOW_GOOGLE_DOMAINvalues below have been replaced with dummy data. You can see we are retrieving Google credentials from a kubernetes secret which is configured via ExternalSecrets. oAuth values were set in the helm chart and the callback URLs were configured on the Google GCP side for the application. Authentication for the web gui washandled via Google oAuth, over HTTPS - a fairly standard setup. Our original implementation of Apache Airflow was deployed onto EKS using the community maintained helm chart. Self-hosted Implementation The original configuration This article will go through the specific issues we faced, and the journey to migrating to mwaa. We spent several days attempting to resolve these problems before deciding that it was better to simply switch to using AWS Managed Workflows for Apache Airflow (mwaa). During the course of the migration we encountered some issues with Apache Airflow around correct traffic routing/authentication for the web gui. The old EKS cluster was using istio as an ingress gateway controller, however we dropped this on the new cluster and opted for a more managed approach of using the AWS Loadbalancer Controller for the majority of ingress, and the nginx ingress controller for any services which required more complex ingress rules. Due to security, and compatibility issues with migrating our self-hosted Airflow envirinment, we decided to migrate to AWS Managed Workflows for Apache Airflow (mwaa).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |