# AWS SQS Queue Implementation - Testing Guide

> **⚠️ New 2-Queue Architecture:** This guide covers the refactored setup using 2 queues instead of 5.
> - **Communication Queue:** Email, WhatsApp, SMS (fast operations)
> - **Heavy Processing Queue:** Invoice PDF generation, Payment receipts (slow operations)

## Test the Setup End-to-End

### Phase 1: Local Development Testing (Without AWS)

1. **Start your application:**
   ```bash
   npm run start:dev
   ```

2. **Verify Queue Module loads:**
   Look for these logs:
   ```
   [QueueModule] Initializing Queue Module...
   [ProcessorRegistryService] Registered processor: EmailProcessor (communication/email)
   [ProcessorRegistryService] Registered processor: WhatsAppProcessor (communication/whatsapp)
   [ProcessorRegistryService] Registered processor: InvoiceProcessor (heavy-processing/invoice)
   [QueueModule] Queue Module initialized successfully
   ```

3. **You should see warnings (this is expected):**
   ```
   [EmailQueueService] Communication queue is disabled or not configured
   [InvoiceQueueService] Heavy processing queue is disabled or not configured
   [SqsPollerService] Queue URL for 'communication' is not configured
   ```

   This is normal - queues are disabled by default.

### Phase 2: Test with Mock AWS Setup (Recommended)

Install LocalStack for local SQS testing:

```bash
# Install LocalStack
pip install localstack

# Start LocalStack with SQS
localstack start

# Create both test queues
aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name test-communication-queue --region ap-south-1
aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name test-heavy-processing-queue --region ap-south-1

# Create DLQs (optional for testing)
aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name test-communication-queue-dlq --region ap-south-1
aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name test-heavy-processing-queue-dlq --region ap-south-1
```

Update your `.env.development`:
```bash
AWS_SQS_REGION=ap-south-1
AWS_SQS_ACCESS_KEY_ID=test
AWS_SQS_SECRET_ACCESS_KEY=test

# New 2-queue setup
AWS_SQS_COMMUNICATION_QUEUE_URL=http://localhost:4566/000000000000/test-communication-queue
AWS_SQS_HEAVY_PROCESSING_QUEUE_URL=http://localhost:4566/000000000000/test-heavy-processing-queue

# Enable queues
ENABLE_COMMUNICATION_QUEUE=true
ENABLE_HEAVY_PROCESSING_QUEUE=true
```

### Phase 3: Integration Test with Real AWS

1. **Create AWS SQS Queues:**
   - Go to AWS Console → SQS
   - Create 4 queues (2 main + 2 DLQ):
     - `dev-infinitheism-communication-queue` (Standard)
     - `dev-infinitheism-communication-queue-dlq` (Standard)
     - `dev-infinitheism-heavy-processing-queue` (Standard)
     - `dev-infinitheism-heavy-processing-queue-dlq` (Standard)
   - Configure DLQ redrive policy (max receive count: 3)
   - Copy the main queue URLs

   **Quick Setup (AWS CLI):**
   ```bash
   # Create DLQs first
   aws sqs create-queue --queue-name dev-infinitheism-communication-queue-dlq --region ap-south-1
   aws sqs create-queue --queue-name dev-infinitheism-heavy-processing-queue-dlq --region ap-south-1
   
   # Get DLQ ARNs (needed for main queues)
   COMM_DLQ_ARN=$(aws sqs get-queue-attributes --queue-url <comm-dlq-url> --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
   HEAVY_DLQ_ARN=$(aws sqs get-queue-attributes --queue-url <heavy-dlq-url> --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)
   
   # Create main queues with DLQ configured
   aws sqs create-queue --queue-name dev-infinitheism-communication-queue \
     --attributes '{"VisibilityTimeout":"300","RedrivePolicy":"{\"deadLetterTargetArn\":\"'$COMM_DLQ_ARN'\",\"maxReceiveCount\":3}"}' \
     --region ap-south-1
     
   aws sqs create-queue --queue-name dev-infinitheism-heavy-processing-queue \
     --attributes '{"VisibilityTimeout":"600","RedrivePolicy":"{\"deadLetterTargetArn\":\"'$HEAVY_DLQ_ARN'\",\"maxReceiveCount\":3}"}' \
     --region ap-south-1
   ```

2. **Configure environment:**
   ```bash
   AWS_SQS_REGION=ap-south-1
   AWS_SQS_ACCESS_KEY_ID=<your-key>
   AWS_SQS_SECRET_ACCESS_KEY=<your-secret>
   
   # New 2-queue setup
   AWS_SQS_COMMUNICATION_QUEUE_URL=<communication-queue-url>
   AWS_SQS_HEAVY_PROCESSING_QUEUE_URL=<heavy-processing-queue-url>
   
   # Enable queues
   ENABLE_COMMUNICATION_QUEUE=true
   ENABLE_HEAVY_PROCESSING_QUEUE=true
   ```

3. **Restart application:**
   ```bash
   npm run start:dev
   ```

4. **Send a test email:**

   Create a test endpoint in any controller:
   ```typescript
   import { EmailQueueService } from './queue/services/email-queue.service';

   @Get('test-queue-email')
   async testQueueEmail() {
     const result = await this.emailQueueService.queueEmail({
       to: {
         emailAddress: 'test@example.com',
         name: 'Test User',
       },
       from: {
         address: 'noreply@infinitheism.org',
         name: 'Infinitheism',
       },
       subject: 'Test Email from Queue',
       templateKey: 'YOUR_TEMPLATE_KEY',
       templateData: {
         userName: 'Test User',
         message: 'This is a test email sent via SQS queue!',
       },
     });

     return { success: true, messageId: result };
   }
   ```

5. **Call the endpoint:**
   ```bash
   curl http://localhost:3000/test-queue-email
   ```

6. **Watch the logs:**
   ```
   [EmailQueueService] Email queued successfully: test@example.com, MessageId: xxx
   [SqsPollerService] Polling communication queue...
   [SqsPollerService] Received 1 message(s) from communication queue
   [EmailProcessor] Processing email message: test@example.com (Attempt: 1)
   [CommunicationService] Sending single email to: test@example.com
   [EmailProcessor] Successfully sent email to: test@example.com in 1234ms
   [SqsPollerService] Successfully processed message from communication queue
   ```

### Phase 4: Verify Queue Processing

1. **Check AWS Console:**
   - Go to your SQS queue
   - Messages available should be 0 (processed successfully)
   - Check CloudWatch metrics

2. **Test failure handling:**
   - Send an email with invalid email address
   - Check logs for retry behavior
   - After 3 attempts, message should move to DLQ (if configured)

3. **Check processing statistics:**
   ```typescript
   // In any controller
   @Get('queue-stats')
   async getQueueStats() {
     return this.sqsPoller.getPollingStats();
   }
   ```

   Response:
   ```json
   {
     "communication": {
       "totalPolled": 10,
       "totalProcessed": 9,
       "totalFailed": 1,
       "successRate": "90.00%",
       "lastPolledAt": "2026-03-23T10:30:00.000Z"
     },
     "heavy-processing": {
       "totalPolled": 5,
       "totalProcessed": 5,
       "totalFailed": 0,
       "successRate": "100.00%",
       "lastPolledAt": "2026-03-23T10:30:00.000Z"
     }
   }
   ```

## Common Issues and Solutions

### Issue 1: Queue Module not polling

**Symptoms:**
- No logs from SqsPollerService
- Messages stay in queue

**Solution:**
- Ensure `ScheduleModule.forRoot()` is in app.module.ts ✅ (already there)
- Check that `ENABLE_COMMUNICATION_QUEUE=true` or `ENABLE_HEAVY_PROCESSING_QUEUE=true`
- Verify queue URLs are correct

### Issue 2: Messages not being deleted

**Symptoms:**
- Messages are processed but reappear in queue
- Same message processed multiple times

**Solution:**
- Check visibility timeout (should be >= 300 seconds)
- Ensure deleteMessage is called after successful processing
- Check logs for deletion errors

### Issue 3: AWS credentials error

**Symptoms:**
```
The security token included in the request is invalid
```

**Solution:**
- Verify AWS_SQS_ACCESS_KEY_ID and AWS_SQS_SECRET_ACCESS_KEY
- Ensure IAM user has sqs:* permissions
- If using IAM role, ensure role is attached to EC2/ECS instance

### Issue 4: Cannot find queue URL

**Symptoms:**
```
AWS.SimpleQueueService.NonExistentQueue
```

**Solution:**
- Verify queue exists in AWS Console
- Check region matches (AWS_SQS_REGION)
- Copy exact URL from AWS Console

## Performance Testing

1. **Load test - Queue 100 emails:**
   ```typescript
   @Get('load-test')
   async loadTest() {
     const promises = [];
     for (let i = 0; i < 100; i++) {
       promises.push(
         this.emailQueueService.queueEmail({
           to: { emailAddress: `test${i}@example.com` },
           from: { address: 'noreply@infinitheism.org', name: 'Test' },
           subject: `Load Test ${i}`,
           templateKey: 'test',
         })
       );
     }
     await Promise.all(promises);
     return { queued: 100 };
   }
   ```

2. **Monitor processing:**
   - Watch CloudWatch metrics
   - Check application logs
   - Monitor API server CPU/memory

3. **Expected behavior:**
   - Queuing 100 emails should take < 5 seconds
   - Processing happens asynchronously over next ~2-3 minutes
   - API remains responsive during processing

## Testing Different Queue Types

### Test Communication Queue (Email, WhatsApp, SMS)

```typescript
// Test Email
await this.emailQueueService.queueEmail({...});

// Test WhatsApp
await this.whatsAppQueueService.queueWhatsAppMessage({...});

// All go to same queue with different subTypes
```

### Test Heavy Processing Queue (Invoice, Payment Receipts)

```typescript
// Test Invoice Generation
await this.invoiceQueueService.queueInvoiceGeneration({...});

// Goes to heavy-processing queue
```

## Next Steps After Successful Testing

✅ 2-Queue architecture implemented  
✅ Email processor working  
✅ WhatsApp processor working  
✅ SMS processor working (if enabled)  
✅ Invoice processor working  
✅ Payment Receipt processor (if needed)  
✅ Integrated into all services  
⬜ Setup CloudWatch alarms for both queues  
⬜ Configure auto-scaling based on queue depth  
⬜ Monitor DLQ for persistent failures