TLDR
AWS Lambda vulnerabilities stem from configuration mistakes, bad programming, and vulnerabilities in dependencies. To manage vulnerabilities, start scanning and patching more.
Foreword
The unpatched, common vulnerability caused some recent data breaches in Finland. This made me think about the security of the cloud solutions I’m working with. This time, I’m concentrating on one of my all-time computing service favorites: AWS Lambda functions. But AWS Lambda is a fully managed service that is serverless, and AWS takes care of patching, right? Yes, but there is also shared responsibility for the security of Lambda functions, and your part relates to vulnerabilities in your code, software dependencies within the code, and the Lambda layers. Not to forget the IAM permissions you give to the function. AWS handles things like compute resources, runtime languages, server software, networking infrastructure, and underlying hardware.
In this blog post, I describe possible threats and exploits and concentrate more on code vulnerabilities and the Amazon Inspector service. I also provide a list of AWS Lambda security best practices.
The Nature of Serverless
Event-driven. AWS Lambda is designed to run code in response to events. When an event occurs, it triggers the Lambda function, passing along event data that the function processes. The triggering event can come from various supported sources, such as REST API, message queue, updated database table, or S3 object being created or modified, or it can be a custom event you send from the application code. This is the input that needs to be validated. AWS Lambda supports a massive list of triggers, a vast list of entry points for possible exploits, and, thus, a massive list of different input event formats. This is a challenge when you are using serverless on a larger scale.
Stateless by nature. The state is created when the service, Firecracker under the hood, provisions the virtual lightweight Lambda container and is destroyed when the service terminates the container. You can’t fully control or depend on a container being reused.
Threats with AWS Lambda functions with vulnerabilities
What could happen if you do not manage vulnerabilities of your Lambda functions?
Configuration Mistake as an Entrypoint
Public serverless web applications are an easy example because anyone can try to exploit HTTP requests. A Web Application Firewall is an excellent safeguard for web application APIs, but it’s incapable of inspecting events from non-HTTP event sources. A classic example is a compromised, open S3 bucket. The attacker can upload something else you might expect or manipulate the object metadata, like tags. The attacker could even use the object name for injections.
Event-Data Injection
AWS Lambda event-data injection is a technique where an attacker manipulates the input data sent to an AWS Lambda function to exploit the function’s behavior. This could lead to unauthorized actions via the temporary AWS credentials available (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) or your secrets in environment variables. This might lead to data leakage or other malicious activities.
The actual injections can be categorized as follows:
- OS Command injection is a security vulnerability when an attacker executes arbitrary operating system commands on a lambda function. An example could be Python’s os.system() method without proper validation and sanitization.
- SQL / NoSQL injections occur when an attacker can manipulate database queries. Malicious database queries can lead to unauthorized access, data theft, data manipulation, or denial of service.
- Function runtime code injections (Python, JavaScript, Go, etc.) are security vulnerabilities in which an attacker can inject and execute arbitrary code within the runtime environment of an AWS Lambda function.
- Cross-site scripting (XSS) is a security vulnerability typically found in web applications. An attacker can inject malicious scripts into content delivered to other users. In AWS Lambda, XSS vulnerabilities can arise if a Lambda function handles web request events, and an attacker can inject malicious code into the Lambda response.
Amazon Inspector
Amazon Inspector is a vulnerability management service that scans your workloads for known software vulnerabilities and unintended network exposures. It creates findings with resource information and severity ratings of the vulnerabilities and offers remediation guidance. It’s yet again one of those security services that supports management in scale with AWS Organizations. So, you can delegate management to a single account within your organization. With the Control Tower landing zone, I recommend the Audit account. Amazon Inspector also integrates with AWS Security Hub for centralized monitoring and alerting. It’s also a recommended checkbox to your production readiness checklist that you check for any Common Vulnerabilities and Exposures (CVE).
Amazon Inspector and AWS Lambda
Amazon Inspector for AWS Lambda functions offers automated, continuous security assessments that identify vulnerabilities in your functions and layers. It offers two different functionalities: Lambda standard scanning, which is also the default, or you can activate standard scanning together with Lambda code scanning.
Lambda Standard Scanning
Lambda standard scanning scans application dependencies within a Lambda function and its layers for dependency package vulnerabilities. For example, if a dependency package contains a function with a vulnerability, Lambda standard scanning will generate a finding.
Lambda Code Scanning
For code vulnerabilities, Lambda code scanning scans custom application code in a Lambda function. Scanning is implemented under the hood with Amazon CodeGuru service. The hype around generative AI is intense. Also, Amazon Inspector got a new feature in Invent 2023, which utilizes Gen AI. It extends the Amazon Inspector capabilities to assess your Lambda code for security issues like injection flaws, data leaks, weak cryptography, or missing encryption based on AWS security best practices. For a list of possible detections, see the CodeGuru Detector Library.
Initiating the Scans
There are a couple of triggers for initiating the vulnerability scans:
- As soon as Amazon Inspector discovers an existing Lambda function.
- When you deploy a new Lambda function.
- When you update the Lambda code or dependencies of existing Lambda functions or their layers.
- Whenever Amazon Inspector adds a new CVE item to its database, and that CVE is relevant to your function.
AWS Lambda Security Best Practices
Lambda functions are essentially just small applications. Common web application best practices are still valid, but some are related to the abovementioned unique nature.
Principle of Least Privilege
Only give the functions the permissions they need to do their jobs. Too open IAM policies could be used as a breakpoint for the account and move laterally. How far can the attacker reach? The attacker could use your account for malicious purposes or get access to sensitive data.
Single Responsibility Principle
Program your Lambda functions to perform one and only one action. It keeps the code more simple and reduces the chance of introducing errors. It also helps to limit the access rights and thus reduce the blast radius should it be compromised.
Validate the Inputs, Block Malicious
Consider all input events potentially harmful. Invalid events should not be processed and should be logged for further analysis.
Monitor Dependencies
Monitor the Lambda function dependencies using Amazon Inspector or third-party/open-source tools to mitigate dependency poisoning attacks.
Review and Scan Your Code
Implement a code review process and additional tooling to find vulnerabilities in your function code. Amazon Inspector and CodeGuru are good for this job, but many third-party tools exist.
Deploy into a VPC
To securely access your private resources, you can apply VPC private networking’s best practices and safeguards, such as routing and firewalls (network ACLs and security groups) or VPC endpoints. You can also get your Lambda networking into the VPC Flow Logs, which can be monitored further.
Protect the Secrets
Use tools like AWS Systems Manager Parameter Store or AWS Secrets Manager to centrally manage the lifecycle of secrets. There is a Lambda layer available that supports both.
HTTP Access
Use Amazon API Gateway or AWS AppSync for HTTP access. Both provide several safeguards and are compatible with the AWS Web Application Firewall (WAF). With WAF, you get features like rate limiting, throttling, and protection against common web exploits and bots.
Clean Up Temporary Files
AWS Lambda has an ephemeral storage, “/tmp” directory, which you can use to write temporary files. The default size is 512 MB, but can be increased to 10,240 MB. Because of the stateless nature described above and container reuse, there is no guarantee that contents will be cleaned up after the invocation. Consider intentionally deleting the contents before the execution completes.
Implement Monitoring
The distributed nature can also make it harder to detect malicious activities. For operational statistics, you should use Amazon CloudWatch or third-party solutions to provide insights about invocations and execution time. Consider using Amazon GuardDuty for threat intelligence and anomaly detection to detect if you’re being attacked.
Conclusion
Even when AWS handles many things, you still need to manage some things. Many traditional web application threats are still valid with serverless web applications. Event-driven apps are not safe either and can be exploited via configuration mistakes, such as unintentional access to an S3 bucket. Review the best practices, and stay safe!
References
Security in AWS Lambda. https://docs.aws.amazon.com/lambda/latest/dg/lambda-security.html
Scanning AWS Lambda functions with Amazon Inspector - Amazon Inspector. https://docs.aws.amazon.com/inspector/latest/user/scanning-lambda.html