Skip to content

Commit b4a1ffa

Browse files
jayeshjehjayeshjeh
authored
Update README.md
1 parent 1c3537e commit b4a1ffa

1 file changed

Lines changed: 165 additions & 31 deletions

File tree

‎README.md‎

Lines changed: 165 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
# Event-Driven Architecture with AWS Lambda and SQS
22

3-
This project demonstrates an event-driven architecture using **AWS Lambda**, **Amazon SQS**, and **AWS CloudWatch** for building a simple event-based processing system. The infrastructure is managed using **Terraform**, and the deployment pipeli## Project Overview
3+
This project demonstrates an event-driven architecture using **AWS Lambda**, **Amazon SQS**, and **AWS CloudWatch** for building a simple event-based processing system. The infrastructure is managed using **Terraform**, and the deployment pipeline is set up using **GitHub Actions** for CI/CD.
4+
5+
---
6+
7+
## Project Overview
48

59
The project contains:
610

@@ -9,6 +13,8 @@ The project contains:
913
- **Terraform Configuration**: Terraform scripts to deploy the Lambda function, SQS queue, and related IAM roles and policies.
1014
- **CI/CD Pipeline**: GitHub Actions pipeline to automatically deploy infrastructure and Lambda code changes.
1115

16+
---
17+
1218
## Architecture
1319

1420
1. **SQS Queue**:
@@ -26,49 +32,177 @@ The project contains:
2632
- GitHub Actions is used for Continuous Integration and Continuous Deployment. It triggers on commits to the `main` branch or when changes are made to infrastructure files (`modules`, `environments`, `lambda_src`).
2733
- Terraform manages infrastructure deployment, and Lambda updates are automatically deployed via GitHub Actions.
2834

29-
## Prerequisitesne is set up using **GitHub Actions** for CI/CD.
35+
---
3036

31-
## Project Overview
37+
## Prerequisites
3238

33-
The project contains:
39+
To get started with this project, make sure you have the following installed:
3440

35-
* **AWS Lambda Function**: A Python-based Lambda function that processes messages from an SQS queue and logs details to CloudWatch.
36-
* **SQS Queue**: An SQS queue that triggers the Lambda function on receiving messages.
37-
* **Terraform Configuration**: Terraform scripts to deploy the Lambda function, SQS queue, and related IAM roles and policies.
38-
* **CI/CD Pipeline**: GitHub Actions pipeline to automatically deploy infrastructure and Lambda code changes.
41+
- **Terraform** (v1.5.5 or later)
42+
- **AWS CLI** (configured with the necessary permissions)
43+
- **GitHub** (to push and manage code)
44+
- **Node.js** (for running the Lambda function locally, if needed)
3945

40-
## Architecture
46+
---
4147

42-
1. **SQS Queue**:
43-
* The SQS queue receives event messages such as "build_succeeded".
44-
* The queue triggers the Lambda function when a new message is pushed.
48+
## Setup
4549

46-
2. **Lambda Function**:
47-
* The Lambda function processes the messages from SQS, logs the message details to CloudWatch, and performs any additional processing (e.g., triggering other services).
48-
* The function is triggered every time a new message is added to the queue.
50+
### 1. Clone the Repository
4951

50-
3. **CloudWatch Logs**:
51-
* The Lambda function logs detailed information about the received message in CloudWatch for monitoring and debugging.
52+
First, clone the repository to your local machine:
5253

53-
4. **CI/CD Pipeline**:
54-
* GitHub Actions is used for Continuous Integration and Continuous Deployment. It triggers on commits to the `main` branch or when changes are made to infrastructure files (`modules`, `environments`, `lambda_src`).
55-
* Terraform manages infrastructure deployment, and Lambda updates are automatically deployed via GitHub Actions.
54+
```bash
55+
git clone https://github.com/<your-username>/Project-Event_driven_Arch.git
56+
cd Project-Event_driven_Arch
57+
```
5658

57-
## Prerequisites
59+
### 2. Terraform Configuration
5860

59-
To get started with this project, make sure you have the following installed:
61+
Navigate to the appropriate environment folder (e.g., `environments/dev`), and initialize Terraform:
6062

61-
* **Terraform** (v1.5.5 or later)
62-
* **AWS CLI** (configured with the necessary permissions)
63-
* **GitHub** (to push and manage code)
64-
* **Node.js** (for running the Lambda function locally, if needed)
63+
```bash
64+
cd environments/dev
65+
terraform init
66+
```
6567

66-
## Setup
68+
### 3. Configure AWS Credentials
6769

68-
### 1. Clone the Repository
70+
You will need AWS credentials with sufficient permissions to manage the Lambda function, SQS, and IAM roles/policies.
6971

70-
First, clone the repository to your local machine:
72+
If you are using GitHub Actions, configure AWS OIDC (OpenID Connect) for authentication. You can store the role ARN in GitHub secrets:
73+
74+
- **AWS OIDC Role**: Create an IAM role in AWS for GitHub Actions using OIDC authentication.
75+
- **GitHub Secrets**: Add `AWS_OIDC_ROLE_ARN` to your GitHub repository secrets with the ARN of the role.
76+
77+
### 4. Deploy Infrastructure Using Terraform
78+
79+
To deploy the infrastructure (SQS queue, Lambda function, IAM roles), run:
7180

7281
```bash
73-
git clone https://github.com/<your-username>/Project-Event_driven_Arch.git
74-
cd Project-Event_driven_Arch
82+
terraform plan -var-file=terraform.tfvars
83+
terraform apply -auto-approve -var-file=terraform.tfvars
84+
```
85+
86+
This will deploy the resources and the Lambda function.
87+
88+
### 5. Configure GitHub Actions CI/CD
89+
90+
The GitHub Actions pipeline is configured to automatically deploy updates when you push code changes to the repository.
91+
92+
In the `.github/workflows` directory, the `terraform.yml` file defines the CI/CD pipeline:
93+
94+
- **Checkout code**
95+
- **Setup Terraform**
96+
- **Configure AWS credentials**
97+
- **Run Terraform commands** (e.g., `fmt`, `init`, `validate`, `plan`, `apply`)
98+
99+
---
100+
101+
## Testing Lambda
102+
103+
You can manually test the Lambda function by sending a message to the SQS queue:
104+
105+
```bash
106+
aws sqs send-message --queue-url <SQS_QUEUE_URL> --message-body '{"type":"build_succeeded","build_id":123}'
107+
```
108+
109+
To view the logs for the Lambda function, use:
110+
111+
```bash
112+
aws logs tail /aws/lambda/ci-event-processor --since 5m --follow
113+
```
114+
115+
---
116+
117+
## Monitor CI/CD Pipeline
118+
119+
After pushing changes to the main branch or opening a pull request, GitHub Actions will automatically trigger the pipeline. You can view the pipeline logs in GitHub under the "Actions" tab.
120+
121+
---
122+
123+
## Manual Testing via Console
124+
125+
- **Lambda Function**: You can test the Lambda function directly in the AWS Lambda console by sending sample event data.
126+
- **SQS Queue**: Messages can be sent to the SQS queue using the AWS CLI or via the AWS Console.
127+
128+
---
129+
130+
## Folder Structure
131+
132+
```
133+
Project-Event_driven_Arch/
134+
├── environments
135+
│   ├── dev
136+
│   │   ├── main.tf
137+
│   │   ├── outputs.tf
138+
│   │   ├── terraform.tfvars
139+
│   │   └── variables.tf
140+
│   └── prod
141+
│   ├── main.tf
142+
│   ├── terraform.tfvars
143+
│   └── variables.tf
144+
├── lambda_src
145+
│   ├── index.py
146+
│   └── lambda.zip
147+
├── modules
148+
│   ├── event_mapping
149+
│   │   ├── main.tf
150+
│   │   ├── outputs.tf
151+
│   │   └── variables.tf
152+
│   ├── iam
153+
│   │   ├── main.tf
154+
│   │   ├── outputs.tf
155+
│   │   └── variables.tf
156+
│   ├── lambda
157+
│   │   ├── main.tf
158+
│   │   ├── outputs.tf
159+
│   │   └── variables.tf
160+
│   ├── sg
161+
│   │   ├── main.tf
162+
│   │   ├── outputs.tf
163+
│   │   └── variables.tf
164+
│   ├── sqs
165+
│   │   ├── main.tf
166+
│   │   ├── outputs.tf
167+
│   │   └── variables.tf
168+
│   └── vpc
169+
│   ├── main.tf
170+
│   ├── outputs.tf
171+
│   └── variables.tf
172+
└── README.md
173+
174+
12 directories, 28 files
175+
176+
```
177+
178+
---
179+
180+
## Troubleshooting
181+
182+
### Common Issues:
183+
184+
- **No CloudWatch logs**: Ensure the Lambda function has the necessary IAM permissions to create and write to CloudWatch Logs.
185+
- **Lambda not triggered**: Verify the SQS queue and Lambda trigger are correctly configured. Ensure the Lambda function is subscribed to the SQS queue.
186+
- **Terraform apply issues**: Ensure that Terraform is configured with the correct AWS credentials, and that you are running it in the correct region.
187+
188+
### Logs:
189+
190+
- **Lambda logs**: Access them in the CloudWatch console or via AWS CLI:
191+
192+
```bash
193+
aws logs tail /aws/lambda/ci-event-processor
194+
```
195+
196+
- **GitHub Actions pipeline logs**: Available in the "Actions" tab in your GitHub repository.
197+
198+
---
199+
200+
## Contributing
201+
202+
Feel free to fork the repository, make changes, and submit pull requests. Please ensure all changes are well-documented and have corresponding tests.
203+
204+
---
205+
206+
## License
207+
208+
This project is licensed under the MIT License - see the LICENSE file for details.

0 commit comments

Comments
 (0)