Enhancing Web Application Security Visibility: Storing AWS WAF Logs with Amazon Kinesis Data Firehose
Customer Name: Airpay Payment Services
[CSP: AWS | Vertical: FinTech | Services Used: Managed Service Provider (MSP)]Client Profile
Airpay is India's first integrated omnichannel financial services platform. Airpay has nearly a decade of experience as a payments aggregator. Their financial services platform is used by Consumers, Businesses, Banks and Financial Institutions from all around the world. They collaborate with over 200+ financial institutions and over 1000 business partners worldwide.
Airpay, a leading fintech company, needed an efficient, scalable, and secure solution to manage and store AWS Web Application Firewall (WAF) logs generated by their applications. These logs were crucial for monitoring security threats, ensuring compliance, and conducting forensic analysis of potential cyberattacks.
Challenge
✓ Delayed Log Availability: The legacy batch-based ingestion caused delays in log availability, making it difficult to track real-time user activity and consumption patterns.
✓ Inability to Handle Fluctuating Log Volumes: Log volumes fluctuated with traffic spikes, and the existing system lacked dynamic scaling, leading to bottlenecks and occasional data loss.
✓ High Operational Complexity: Managing a self-hosted logging setup required continuous monitoring, scaling, and maintenance, increasing operational overhead and resource effort.
Solution
To address these challenges, we proposed AWS Kinesis Data Firehose as a fully managed, scalable, and cost-effective solution for streaming and storing WAF logs in Amazon S3. Delivered through our cloud consulting services, the solution enabled real-time log ingestion, automatic scaling, and seamless integration with other AWS services. Ongoing operations were supported through our managed services and security and compliance services, with continuous threat monitoring and incident response handled by our SOC Services, and 24/7 infrastructure and traffic monitoring ensured by our NOC Services.
AWS MSP Services
✓ AWS Kinesis Data Firehose: To store WAF logs
✓ AWS IAM: For securely managing access to AWS services and resources
✓ Amazon CloudWatch: To monitor applications and services and gain actionable insights
✓ AWS CloudTrail: To log, continuously monitor, and retain account activity across AWS resources
✓ AWS CloudFront: To distribute content globally with low latency
✓ AWS EC2: Provides scalable compute capacity to run applications
✓ AWS S3: For storage purposes
✓ AWS Athena: To run SQL queries directly on logs stored in S3 without needing a dedicated database or analytics engine
Business Impact
✓ Enhanced Real-Time Log Availability: Log delivery time is reduced from hours to real-time, enabling the security team to identify and respond to threats instantly.
✓ Seamless Auto-Scaling During Traffic Surges: Kinesis Data Firehose automatically handled traffic spikes, ensuring reliable, lossless data ingestion at all times.
✓ Simplified Log Storage and Management: Logs were securely stored in Amazon S3, providing scalable, durable, and cost-effective storage for all log data.
✓ On-Demand Log Analytics Without Pipelines: Amazon Athena enabled direct querying of logs in S3, delivering actionable insights in real-time without additional ETL processes.
✓ Improved Operational Efficiency and Threat Response: The integrated, serverless architecture, supported by our managed services, streamlined log management, reducing operational overhead and accelerating security operations.
