# Queue Module Environment Configuration

## Simplified 2-Queue Architecture

This implementation uses **2 queues** instead of 5 for simplicity:

1. **Communication Queue** - Handles all user-facing communications (email, WhatsApp, SMS)
2. **Heavy Processing Queue** - Handles CPU-intensive tasks (invoice generation, payment receipts)

### Why 2 Queues?

✅ **Simpler configuration** - Only 2 queue URLs, 2 feature flags  
✅ **Lower AWS cost** - ~60% cheaper than 5 queues  
✅ **Matches your usage** - You send email + WhatsApp together anyway  
✅ **Different processing times** - Communication is fast (~1-3s), invoice is slow (~10-30s)  
✅ **Easier monitoring** - 2 CloudWatch dashboards instead of 5  

## Required Environment Variables

Add these variables to your environment files:
- `environment/.env.development`
- `environment/.env.production`
- `environment/.env.staging`

```bash
# ============================================
# AWS SQS Configuration
# ============================================

# AWS Region where your SQS queues are created
AWS_SQS_REGION=ap-south-1

# AWS Credentials (Optional if using IAM roles)
# If running on EC2/ECS with IAM role, these can be omitted
AWS_SQS_ACCESS_KEY_ID=your_access_key_here
AWS_SQS_SECRET_ACCESS_KEY=your_secret_key_here

# ============================================
# SQS Queue URLs (2 queues only)
# ============================================
# You need to create these queues in AWS SQS first
# Queue naming convention: <env>-infinitheism-<queue-type>-queue

# Communication Queue - Handles: Email, WhatsApp, SMS
AWS_SQS_COMMUNICATION_QUEUE_URL=https://sqs.ap-south-1.amazonaws.com/123456789/prod-infinitheism-communication-queue

# Heavy Processing Queue - Handles: Invoice generation, Payment receipts
AWS_SQS_HEAVY_PROCESSING_QUEUE_URL=https://sqs.ap-south-1.amazonaws.com/123456789/prod-infinitheism-heavy-processing-queue

# ============================================
# Queue Feature Flags (Enable/Disable Queues)
# ============================================
# Set to 'true' to enable queue processing
# Set to 'false' to disable (falls back to synchronous processing)

ENABLE_COMMUNICATION_QUEUE=false
ENABLE_HEAVY_PROCESSING_QUEUE=false
```

## Creating SQS Queues in AWS

You have two options to create the queues: AWS Console (GUI) or AWS CLI (command line).

### Option 1: AWS Console (Web Interface)

#### Step 1: Create Dead Letter Queues First

**Communication DLQ:**
1. Go to [AWS SQS Console](https://console.aws.amazon.com/sqs/)
2. Click **"Create queue"**
3. Select **"Standard"** queue type
4. Queue name: `prod-infinitheism-communication-dlq`
5. Configuration → Keep defaults
6. Click **"Create queue"**

**Heavy Processing DLQ:**
1. Click **"Create queue"**
2. Select **"Standard"** queue type
3. Queue name: `prod-infinitheism-heavy-processing-dlq`
4. Configuration → Keep defaults
5. Click **"Create queue"**

#### Step 2: Create Main Queues with DLQ Attached

**Communication Queue:**
1. Click **"Create queue"**
2. Select **"Standard"** queue type
3. Queue name: `prod-infinitheism-communication-queue`
4. Configuration:
   - **Visibility timeout:** `300` seconds (5 minutes)
   - **Message retention period:** `1209600` seconds (14 days)
   - **Delivery delay:** `0` seconds
   - **Maximum message size:** `256` KB
   - **Receive message wait time:** `20` seconds (enables long polling)
5. **Dead-letter queue:**
   - ✅ Enable
   - Choose: `prod-infinitheism-communication-dlq`
   - Maximum receives: `3`
6. Click **"Create queue"**
7. **Copy the Queue URL** (needed for `.env` file)

**Heavy Processing Queue:**
1. Click **"Create queue"**
2. Select **"Standard"** queue type
3. Queue name: `prod-infinitheism-heavy-processing-queue`
4. Configuration:
   - **Visibility timeout:** `600` seconds (10 minutes - longer for PDF generation)
   - **Message retention period:** `1209600` seconds (14 days)
   - **Delivery delay:** `0` seconds
   - **Maximum message size:** `256` KB
   - **Receive message wait time:** `20` seconds (enables long polling)
5. **Dead-letter queue:**
   - ✅ Enable
   - Choose: `prod-infinitheism-heavy-processing-dlq`
   - Maximum receives: `3`
6. Click **"Create queue"**
7. **Copy the Queue URL** (needed for `.env` file)

---

### Option 2: AWS CLI (Command Line)

**Prerequisites:**
- Install [AWS CLI](https://aws.amazon.com/cli/)
- Configure credentials: `aws configure`
- Set your region: `ap-south-1` (or your preferred region)

**Complete Setup Script:**

```bash
#!/bin/bash
# SQS Queue Setup Script for 2-Queue Architecture
# Replace 'prod' with your environment (dev, staging, prod)

ENV_PREFIX="prod"
REGION="ap-south-1"

echo "Creating Dead Letter Queues..."

# Step 1: Create Communication DLQ
COMM_DLQ_URL=$(aws sqs create-queue \
  --queue-name ${ENV_PREFIX}-infinitheism-communication-dlq \
  --region ${REGION} \
  --query 'QueueUrl' \
  --output text)

echo "Communication DLQ created: $COMM_DLQ_URL"

# Get Communication DLQ ARN
COMM_DLQ_ARN=$(aws sqs get-queue-attributes \
  --queue-url ${COMM_DLQ_URL} \
  --attribute-names QueueArn \
  --region ${REGION} \
  --query 'Attributes.QueueArn' \
  --output text)

echo "Communication DLQ ARN: $COMM_DLQ_ARN"

# Step 2: Create Heavy Processing DLQ
HEAVY_DLQ_URL=$(aws sqs create-queue \
  --queue-name ${ENV_PREFIX}-infinitheism-heavy-processing-dlq \
  --region ${REGION} \
  --query 'QueueUrl' \
  --output text)

echo "Heavy Processing DLQ created: $HEAVY_DLQ_URL"

# Get Heavy Processing DLQ ARN
HEAVY_DLQ_ARN=$(aws sqs get-queue-attributes \
  --queue-url ${HEAVY_DLQ_URL} \
  --attribute-names QueueArn \
  --region ${REGION} \
  --query 'Attributes.QueueArn' \
  --output text)

echo "Heavy Processing DLQ ARN: $HEAVY_DLQ_ARN"

echo "Creating Main Queues..."

# Step 3: Create Communication Queue with DLQ
COMM_QUEUE_URL=$(aws sqs create-queue \
  --queue-name ${ENV_PREFIX}-infinitheism-communication-queue \
  --region ${REGION} \
  --attributes "{
    \"VisibilityTimeout\": \"300\",
    \"MessageRetentionPeriod\": \"1209600\",
    \"ReceiveMessageWaitTimeSeconds\": \"20\",
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${COMM_DLQ_ARN}\\\",\\\"maxReceiveCount\\\":3}\"
  }" \
  --query 'QueueUrl' \
  --output text)

echo "Communication Queue created: $COMM_QUEUE_URL"

# Step 4: Create Heavy Processing Queue with DLQ
HEAVY_QUEUE_URL=$(aws sqs create-queue \
  --queue-name ${ENV_PREFIX}-infinitheism-heavy-processing-queue \
  --region ${REGION} \
  --attributes "{
    \"VisibilityTimeout\": \"600\",
    \"MessageRetentionPeriod\": \"1209600\",
    \"ReceiveMessageWaitTimeSeconds\": \"20\",
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${HEAVY_DLQ_ARN}\\\",\\\"maxReceiveCount\\\":3}\"
  }" \
  --query 'QueueUrl' \
  --output text)

echo "Heavy Processing Queue created: $HEAVY_QUEUE_URL"

# Step 5: Output summary
echo ""
echo "========================================="
echo "Queue Setup Complete!"
echo "========================================="
echo ""
echo "Add these to your .env file:"
echo ""
echo "AWS_SQS_COMMUNICATION_QUEUE_URL=${COMM_QUEUE_URL}"
echo "AWS_SQS_HEAVY_PROCESSING_QUEUE_URL=${HEAVY_QUEUE_URL}"
echo ""
echo "ENABLE_COMMUNICATION_QUEUE=false"
echo "ENABLE_HEAVY_PROCESSING_QUEUE=false"
echo ""
echo "Set flags to 'true' when ready to enable queues"
```

**To run the script:**
```bash
chmod +x create-queues.sh
./create-queues.sh
```

---

## IAM Permissions Required

Your AWS IAM user or role needs these permissions to send/receive/delete messages:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InfinitheismQueueAccess",
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl"
      ],
      "Resource": [
        "arn:aws:sqs:ap-south-1:*:prod-infinitheism-communication-queue",
        "arn:aws:sqs:ap-south-1:*:prod-infinitheism-communication-dlq",
        "arn:aws:sqs:ap-south-1:*:prod-infinitheism-heavy-processing-queue",
        "arn:aws:sqs:ap-south-1:*:prod-infinitheism-heavy-processing-dlq"
      ]
    }
  ]
}
```

**To create the IAM policy:**
1. Go to [IAM Console](https://console.aws.amazon.com/iam/)
2. Policies → **Create policy**
3. Choose **JSON** tab
4. Paste the policy above (replace `ap-south-1` with your region if different)
5. Name: `InfinitheismSQSAccess`
6. Attach to your IAM user/role

---

## Testing Configuration

Follow these steps to safely enable queues:

**Step 1: Verify Environment Variables**
```bash
# Check that queue URLs are set correctly
echo $AWS_SQS_COMMUNICATION_QUEUE_URL
echo $AWS_SQS_HEAVY_PROCESSING_QUEUE_URL
```

**Step 2: Start with Queues Disabled**
```bash
# In your .env file
ENABLE_COMMUNICATION_QUEUE=false
ENABLE_HEAVY_PROCESSING_QUEUE=false
```

**Step 3: Start Your Application**
```bash
npm run start:dev
```

**Step 4: Check Logs**
Look for: `Queue Module initialized successfully`

**Step 5: Enable Communication Queue**
```bash
# In your .env file
ENABLE_COMMUNICATION_QUEUE=true
```

**Step 6: Test Communications**
- Send a test email
- Send a test WhatsApp message
- Check AWS SQS Console → Queue should show messages processed
- Check application logs for: `Successfully sent email/WhatsApp message`

**Step 7: Enable Heavy Processing Queue (Optional)**
```bash
# In your .env file
ENABLE_HEAVY_PROCESSING_QUEUE=true
```

**Step 8: Test Invoice Generation**
- Generate a test invoice
- Check AWS SQS Console → Queue should show message processed
- Check application logs for: `Successfully generated invoice`

---

## Rollback Plan

If something goes wrong, you can instantly rollback:

1. **Set both feature flags to `false`:**
   ```bash
   ENABLE_COMMUNICATION_QUEUE=false
   ENABLE_HEAVY_PROCESSING_QUEUE=false
   ```

2. **Restart your application:**
   ```bash
   npm run start:dev
   ```

3. **Result:** All processing returns to synchronous mode
4. **No code changes needed!**

---

## Monitoring Queue Health

The Queue Module provides statistics via the SqsPollerService:

```typescript
// In any service:
constructor(private sqsPoller: SqsPollerService) {}

const stats = this.sqsPoller.getPollingStats();
// Returns statistics for each queue type
```

---

## Queue Architecture Details

### Communication Queue
- **Purpose:** Fast user-facing communications
- **Handles:** Email, WhatsApp, SMS
- **Processing time:** 1-3 seconds per message
- **Visibility timeout:** 5 minutes
- **Polling frequency:** Every 10 seconds
- **Concurrency:** 10 messages per poll

### Heavy Processing Queue
- **Purpose:** CPU-intensive background tasks
- **Handles:** Invoice PDF generation, Payment receipts
- **Processing time:** 10-30 seconds per message
- **Visibility timeout:** 10 minutes (longer for PDF generation)
- **Polling frequency:** Every 10 seconds
- **Concurrency:** 10 messages per poll

---

## Message Flow Diagrams

**Communication Queue Flow:**
```
Registration Service / Payment Service
  ↓ queues message
Communication Queue (SQS)
  ↓ polls every 10s
Queue Processor (WhatsApp/Email/SMS)
  ↓ delegates to
CommunicationService / Msg91Service
  ↓ sends via
WATI API / SMTP / MSG91
```

**Heavy Processing Queue Flow:**
```
Payment Service / Invoice Service
  ↓ queues message
Heavy Processing Queue (SQS)
  ↓ polls every 10s
Queue Processor (Invoice/Payment Receipt)
  ↓ delegates to
InvoiceService (Puppeteer PDF generation)
  ↓ sends via
CommunicationService (Email with attachment)
```

---

## Next Steps (Implementation Checklist)

- [x] Queue Module implemented
- [x] Queue services created (WhatsApp, Email, SMS, Invoice)
- [x] Queue processors created
- [ ] **Create SQS queues in AWS** (use guide above)
- [ ] **Configure environment variables** (add queue URLs)
- [ ] **Set feature flags to `false`** initially
- [ ] **Deploy to development environment**
- [ ] **Enable communication queue:** `ENABLE_COMMUNICATION_QUEUE=true`
- [ ] **Test email and WhatsApp sending**
- [ ] **Enable heavy processing queue:** `ENABLE_HEAVY_PROCESSING_QUEUE=true`
- [ ] **Test invoice generation**
- [ ] **Monitor AWS SQS Console** for message flow
- [ ] **Monitor application logs** for errors
- [ ] **Gradually roll out to staging**
- [ ] **Gradually roll out to production**

---

## Cost Estimation

**2 Queues (Communication + Heavy Processing):**
- 1 million requests/month: **$0.40**
- 10 million requests/month: **$4.00**
- First 1 million requests are free tier

**For comparison:**
- Old 5-queue setup would cost ~$10/month at same volume
- **Savings: ~60%**

---

## Troubleshooting

**Messages not being processed?**
- Check feature flags are `true`
- Verify queue URLs are correct
- Check AWS SQS Console → Messages should appear
- Check application logs for errors

**Messages going to Dead Letter Queue?**
- Check DLQ in AWS Console
- Review error logs for failed processing
- Fix the issue
- Manually re-drive messages from DLQ to main queue

**Queue polling not starting?**
- Check logs for "Starting poller for queue type: ..."
- Verify `QueueModule` is imported in `AppModule`
- Check environment variables are loaded correctly