CI/CD Pipeline Q&A

Interview Answer: When a developer raises a Pull Request, the CI pipeline is triggered automatically. It runs code quality checks, unit tests, SQL validation, PySpark tests, and Airflow DAG validation. If all tests pass and the PR is approved, the code is merged into the main branch and becomes ready for deployment.

Interview Answer: Separate environments help us test code safely before releasing it to users. Developers make changes in Dev, testers validate them in QA, and only approved code is deployed to Production. This reduces the risk of production failures and improves software quality.

Interview Answer: If unit tests fail, the CI pipeline stops immediately and the code is not deployed further. The developer checks the test logs, fixes the issue, and pushes the updated code. The pipeline is then triggered again until all tests pass.

Interview Answer: Secrets such as passwords, API keys, and database credentials should never be stored in Git because they can be exposed to unauthorized users. Instead, we use AWS Secrets Manager, Parameter Store, or Azure Key Vault to securely manage sensitive information.

Interview Answer: Infrastructure as Code allows us to create and manage cloud resources using code instead of manually creating them. Tools like Terraform and AWS CDK help automate infrastructure deployment. This provides consistency, version control, and faster deployments.

Interview Answer: We use the same application code across all environments and change only the configuration values. Parameters such as S3 buckets, database names, IAM roles, and secrets are stored separately for Dev, QA, and Production. This makes deployments safer and easier to maintain.

Interview Answer: Approval gates ensure that production deployment happens only after manual approval. Before approving, we verify test results, deployment logs, and release notes. This reduces the risk of deploying incorrect or unstable code to production.

Interview Answer: Before deployment, I make sure all unit tests, integration tests, and data quality checks have passed. I also validate the deployment in Dev and QA environments. If required, I keep a rollback plan ready in case any production issue occurs.

Interview Answer: After deployment, I monitor CloudWatch logs, Glue job status, Airflow DAG runs, execution time, and failure alerts. I also compare source and target record counts to verify successful data processing. Any failures are immediately notified to the support team.

Interview Answer: Common production issues include failed unit tests, missing Python packages, IAM permission errors, broken Airflow DAGs, schema mismatches, and incorrect environment variables. I troubleshoot these issues using pipeline logs, fix the root cause, and redeploy the application.

Interview Answer: Git is used as the version control system to manage source code. Developers create feature branches, commit changes, and raise Pull Requests. Whenever code is merged into the main branch, the CI/CD pipeline is automatically triggered for testing and deployment.

Interview Answer: I reduce deployment failures by using automated testing, code reviews, environment-specific configurations, and proper secret management. I also validate deployments in Dev and QA before moving them to Production. This ensures stable and reliable deployments.