I am trying to convert a JSON file to CSV using Lamda. I am using Pandas for this operation. Initially I started with the following configuration :
File Size : 5 MB
Memory : 128
It took me around 5 seconds to complete the conversion.
Then I increased the file size to 10 MB, but there is a weird behavior. It will be great if someone could help me to understand this. Basically I am trying to benchmark this operation Sometimes the file is getting processed successfully and sometimes it is getting timeout with message
REPORT RequestId: 28e55591-e6a7-4344-b5bc-321bd03422b6 Duration: 900089.03 ms Billed Duration: 900000 ms Memory Size: 128 MB Max Memory Used: 129 MB
It can be clearly seen that this a memory issue, but I am not able to understand the root cause. It will be great if someone could help me to understand this behavior.