AWS CodePipeline and CI/CD

A CI/CD pipeline is a production system that changes other production systems. A green pipeline is useful only when it proves the right source was built, the intended artifact was tested, and that exact artifact was deployed to the correct environment. AWS CodePipeline orchestrates delivery stages, while CodeBuild and deployment services perform work within those stages. Reliability depends on artifact integrity, permission boundaries, environment separation, test quality, and a recovery plan—not simply on adding more stages to a visual workflow.

What You Will Learn

• How source, build, test, approval and deploy stages fit together
• CodePipeline, CodeBuild and deployment integrations
• Artifact flow, permissions and environment separation
• Safe promotion, rollback and release strategies
• Troubleshooting pipeline failures
• Production controls for secure continuous delivery

At a Glance

A Practical Architecture

Source repository
  ↓ webhook / source action
CodePipeline
  ├─ build + unit tests (CodeBuild)
  ├─ security / quality gates
  ├─ approval for high-risk releases
  └─ deploy to test → staging → production
       ↓
health checks + metrics + rollback decision

1. The Delivery Pipeline Model

A pipeline usually begins with a source revision, builds and tests an artifact, applies quality or security checks, and promotes the artifact through environments. CodePipeline coordinates actions; CodeBuild can compile code, run tests, build images, and publish outputs; deployment actions may target services such as ECS, Lambda, CloudFormation, or other supported destinations. Keep the responsibility of each stage clear. If a stage only prints a message but does not validate a meaningful condition, it creates the appearance of safety without reducing risk.

2. Build Once, Promote the Same Artifact

A common release defect is rebuilding separately for staging and production. The source may be the same, but dependency resolution, build-time variables, or base images can change the output. Build a versioned artifact once, test it, and promote that same artifact. Record the source commit, artifact digest, build run, and deployment version. Environment-specific configuration should be injected in a controlled way at deployment/runtime rather than silently changing the application binary between environments.

3. Source Triggers and Branch Strategy

Pipeline triggers should match the repository’s release model. A pull-request validation pipeline may run tests and static analysis, while a protected branch or release tag triggers a deployment pipeline. Avoid allowing untrusted pull-request code to access production credentials or deployment roles. Restrict who can change pipeline definitions and build specifications. A malicious change to the pipeline can be as dangerous as a malicious change to application code because it can alter what is built, what is approved, or where credentials are used.

4. CodeBuild Environments and buildspec

A buildspec defines commands, phases, artifacts, and caching behavior for CodeBuild. Keep commands deterministic, fail when tests fail, and make required artifacts explicit. Pin toolchain and dependency versions where practical. Caching can reduce build time, but stale or untrusted cache content must not influence release correctness. Do not echo secrets into logs. Build environments should use short-lived roles and should not inherit broad permissions merely because a build needs to upload one artifact.

5. Artifacts, Encryption and Integrity

Pipeline artifacts are passed between actions and may be stored in S3 or other service-managed locations. Apply appropriate encryption, access control, retention, and separation for sensitive artifacts. Ensure only intended roles can read or overwrite release artifacts. For container deployments, record the immutable image digest rather than relying only on a mutable tag such as `latest`. If the artifact is overwritten after testing, the pipeline may deploy content that was never reviewed. Artifact integrity is a release control, not just a storage concern.

6. Quality and Security Gates

Useful gates include unit and integration tests, static analysis, dependency and container scanning, infrastructure validation, and policy checks. Tune them so failures are actionable; an alert flood that developers routinely ignore is not a meaningful gate. Critical findings should block promotion according to a documented policy, while accepted exceptions should have an owner and expiry. Do not rely on tests alone for security: review IAM changes, secrets handling, network exposure, and deployment role permissions as part of the release.

7. Environment Promotion and Approvals

Use separate accounts or strong boundaries for development, staging, and production. The production deployment role should not be freely available to every build job. Approval steps can be appropriate for high-impact changes, but approvals should present the exact version, change summary, test evidence, and rollback plan. An approval that says only “Deploy?” is weak because the reviewer lacks context. Automate low-risk deployments where confidence is high, and reserve manual gates for risk decisions that genuinely benefit from human review.

8. Deployment Strategies and Health Checks

The right strategy depends on the target service and application. Rolling deployments, blue/green releases, canary traffic, and Lambda aliases each have different failure modes. A deployment action reporting success may mean resources were updated, not that customers can complete a business transaction. Add smoke tests and monitor latency, error rates, saturation, and key business signals after rollout. Define the conditions that trigger pause or rollback before an incident occurs, rather than inventing thresholds while the service is already failing.

9. IAM and Pipeline Security

Separate the pipeline service role, build role, deployment role, and application runtime role where practical. Grant each only the actions and resources required for its stage. Protect `iam:PassRole`, artifact buckets, encryption keys, connection configuration, and pipeline-edit permissions. Cross-account deployments should use deliberate role trust and narrowly scoped access. A build that runs untrusted code should never receive credentials that can modify production infrastructure. Review logs and environment variables for accidental secret exposure.

10. Example: Application Release Flow

A typical service release might validate a pull request, build a container, run unit tests and vulnerability scanning, push the image to ECR, deploy to a test environment, run smoke tests, and then promote the same image digest to production. For a production change, the pipeline can create a change set or deployment plan, request approval, execute the release, and verify health. If smoke tests fail, stop promotion and invoke the defined rollback or traffic-shift procedure. Capture the artifact digest and deployment result for incident analysis.

11. Pipeline Observability and Ownership

Pipeline metrics should show duration, failure rate, stage-level bottlenecks, queue delays, and deployment frequency. Notifications should identify the failed stage and link to logs, rather than merely saying “pipeline failed.” Assign an owner for flaky tests and repeated failures. A pipeline with permanently ignored failures is effectively bypassed. Keep operational runbooks for expired source connections, exhausted concurrency, missing artifacts, and failed deployment actions. Test the pipeline itself periodically, including the path used to roll back a bad release.

12. Common Failures and Diagnosis

A source stage may fail because a connection was revoked, permissions changed, or a webhook did not trigger. CodeBuild may fail from dependency outages, test failures, insufficient compute, disk limits, or missing environment variables. Deployment can fail because the role lacks permissions, a resource is in an unexpected state, or the target service rejects the artifact. Start with the first failed action and its detailed logs. Distinguish code defects from infrastructure or credential failures; rerunning a deterministic failing build without a change usually only consumes time.

13. Advantages, Limits and Trade-offs

CodePipeline provides managed orchestration and integrates with AWS build and deployment services. It does not decide what “safe” means for a particular application. Complex branching and many cross-account actions can make pipeline behavior hard to reason about. Teams should optimize for traceable artifact flow, clear ownership, least privilege, fast feedback, and reliable recovery. A smaller pipeline with meaningful checks is usually better than a long workflow full of redundant stages.

14. Experienced-Engineer Review Checklist

Verify the source revision is trusted; untrusted code cannot access deployment credentials; build inputs are controlled; tests and security gates are meaningful; artifacts are immutable or digest-addressed; environment boundaries are enforced; roles are least-privilege; approvals show the exact change; post-deploy health is checked; and rollback is rehearsed. Ask whether the team can answer, within minutes, which commit and artifact are running in production and how to return to the previous healthy version.

Advantages

When applied with clear ownership and guardrails, this approach improves repeatability, reviewability, operational speed, and audit evidence. It reduces dependence on undocumented manual steps and helps teams diagnose changes using consistent artifacts and records.

Limitations and Trade-offs

Managed services and automation do not eliminate the need for architecture decisions. Permissions, quotas, network paths, resource lifecycle behavior, external dependencies, and human processes can still cause failure. More controls add maintenance overhead, so choose controls proportionate to the blast radius and criticality of the workload.

Common Production Mistakes

Common mistakes include using broad permissions to avoid diagnosing access issues, relying on mutable identifiers, assuming a successful command proves application health, ignoring partial completion, and leaving no owner for recurring jobs. Another frequent failure is changing several layers at once, which makes it difficult to isolate the root cause. Prefer small, reviewable changes and preserve the evidence needed to explain the result.

Practical Review Checklist

• Confirm the target account, region, environment and identity before making changes.
• Use least-privilege permissions and keep secrets out of code, images and logs.
• Review the planned change and identify replacements, deletions or irreversible side effects.
• Handle pagination, retries, timeouts and partial failures deliberately.
• Verify the resulting state and application health after execution.
• Keep logs, metrics, audit records, ownership and a tested recovery path.

15. Production Scenario: A Green Build, Broken Release

A pipeline builds and tests an application successfully, but users receive errors immediately after deployment. The build may have tested the wrong artifact, the production configuration may be missing, a database migration may not be backward compatible, or the deployment action may have completed before the service became healthy. Investigate the deployed artifact digest and compare it with the tested artifact. Check configuration differences, migration logs, health checks, and the first customer-facing failure. Do not start by rerunning the build if the failure occurs only after promotion.

The pipeline should stop promotion when post-deployment verification fails. If rollback is safe, route traffic back to the previous healthy version; if a schema change is irreversible, a forward fix or explicit recovery may be required. Release design should therefore include compatibility across application versions, migration sequencing, and a defined decision point for rollback. Record the release timeline and artifact identity so incident responders can quickly establish what changed.

Frequently Asked Questions

Does a successful deployment action prove the application works?

No. Add smoke tests and monitor runtime and business-level health after deployment.

Should production be rebuilt from source?

Prefer promoting the same tested artifact to preserve artifact integrity.

Why separate build and deployment roles?

It limits the damage a compromised or untrusted build can cause and makes permissions easier to audit.

When should manual approval be used?

Use it for meaningful risk decisions, with artifact identity, change evidence, and rollback details available to the reviewer.

Conclusion

A reliable CI/CD workflow protects the path from source to production. Build once, test the artifact that will be deployed, isolate credentials, enforce meaningful gates, and verify application health after rollout. Treat the pipeline as critical infrastructure with its own monitoring, ownership, security review, and recovery plan.

Canonical URL: https://kodestree.com/aws-codepipeline-and-ci-cd/