java.io.IOException: Closed #7
Description
I have an AWS Lambda function running in us-east-1 that occasionally fails to connect to GitHub's HTTP API. It most often fails on the first API call, which is the endpoint /repos/SpeakEZ/speakez-rails/pulls. After that failure, the AWS Lambda function halts.
START RequestId: ed8e9386-4c53-11e7-8916-3be6d53816d3 Version: $LATEST
14:08:21.199 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO prs.Main$:109 - Starting aws-gh-prs 0.1-SNAPSHOT
14:08:21.199 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO prs.Main$:111 - Inspecting SNS notification...
14:08:21.242 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO prs.Main$:120 - Processed 1 event(s)
14:08:21.260 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO prs.Main$:165 - Pull request was SpeakEZ/master..ashawley/CAP-2052
14:08:21.260 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO prs.Main$:239 - Querying GitHub for open pull request(s)...
14:08:21.261 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] DEBUG com.ning.http.client.AsyncCompletionHandler:60 - Closed
14:08:21.358 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] ERROR prs.Main$:233 - Failed for unexpected reason
java.io.IOException: Closed
at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:98)
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:87)
at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:506)
at com.ning.http.client.AsyncHttpClient$BoundRequestBuilder.execute(AsyncHttpClient.java:225)
at codecheck.github.transport.asynchttp19.AsyncHttp19Request.execute(AsyncHttp19Transport.scala:41)
at codecheck.github.api.GitHubAPI.exec(GitHubAPI.scala:66)
at codecheck.github.operations.PullRequestOp$class.listPullRequests(PullRequestOp.scala:28)
at codecheck.github.api.GitHubAPI.listPullRequests(GitHubAPI.scala:21)
at prs.Main$.mergeFor(Main.scala:247)
at prs.Main$$anonfun$2$$anonfun$5.apply(Main.scala:175)
at prs.Main$$anonfun$2$$anonfun$5.apply(Main.scala:172)
at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682)
at prs.Main$$anonfun$2.apply(Main.scala:172)
at prs.Main$$anonfun$2.apply(Main.scala:106)
at scala.util.Try$.apply(Try.scala:192)
at prs.Main$.handler(Main.scala:106)
at prs.Main.handler(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:456)
at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:375)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:1139)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:278)
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:62)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)
END RequestId: ed8e9386-4c53-11e7-8916-3be6d53816d3
REPORT RequestId: ed8e9386-4c53-11e7-8916-3be6d53816d3 Duration: 1134.79 ms
Billed Duration: 1200 ms Memory Size: 384 MB Max Memory Used: 384 MB
It seems the HTTP client is not retrying the request or waiting on a timeout. It is pretty sudden. I've inspected the source code of the Ning/AHC library:
For some reason the connection is in a closed state right away.
Coincidentally, AWS Lambda will re-run a function if it fails. Typically it will retry twice. However, the subsequent tries by AWS Lambda doesn't usually improve anything.