갑자기 AWS 인스턴스 재부팅 됨 (Suddenly AWS instance was rebooted)

운영중이던 Tomcat 이 갑자기 Exit 되었다.

 

Mar 16, 2020 6:12:58 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
 
 
...
Mar 16, 2020 6:13:05 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
WARNING: The web application [ROOT] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 16, 2020 6:13:05 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [commons-pool-evictor-thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)

 

톰켓은 Docker container 로 운영중이였다.

 

 

docker ps -a

Exited (143) 53 minutes ago                app_app_1

 

143 Code 는 "Exit code 143 corresponds to SIGTERM" OS 에 의해서 컨테이너가 종료되었을때 남는 코드로 보였다.

 

143 Code 를 확인하고 audit.log 와 sys.log 를 확인해보니 OS 에서 종료 SIGNAL 을 발생시킨 로그가 보였다.

 

sys.log

Mar 16 18:07:09 ip-10-230-255-9 systemd[1]: Stopping Docker compose application start...

audit.log

type=SERVICE_STOP msg=audit(1584349629.667:2676): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=dockerd-proxy comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

 

OS 에서 왜 docker container 를 종료하였는지 찾기 시작하였다. 그러던 중 uptime 명령어를 통해서 OS 부팅 시간을 화인해보니 에러가 톰켓이 동작하지 않기 시작한 시점과 일치하는 것을 확인하였다.

root@ip-10-211-50-138:/# last reboot | head -n 1
reboot   system boot  4.4.0-140-generi Mon Mar 16 18:07   still running

root@ip-10-211-50-138:/# uptime
23:13:01 up  5:05,  1 user,  load average: 0.00, 0.00, 0.00

 

즉, OS 가 재부팅되면서 docker container 가 OS 에 의해서 SIGTERM 을 받았던 것이다.

 

무엇이 OS 를 재부팅 시켰는지 확인하기 위하여 AWS Case open 을 해보았다.

 

Moving forward I checked the API call related to instance {instanceId}
I found that "CreateImage" API call was executed on this instance which is generally used to create Amazon Machine Image(AMI) form the instance.
This API call was initiated at  2020-03-16 09:07:09 UTC which is 2020-03-16 18:07 KST local time by IAM user "{userId}" from source IP address "{IP}" via AWS console.
Generally while creating an AMI from instance, by default an instance is rebooted.
This is done to maintain data consistency in the resulting AMI.
However we do provide an option of "NoReboot".
If this option is checked, the instance will not be rebooted.

 

AMI 생성 요청에 의해서 Instance 가 재부팅 되었으며 AMI 생성 요청에 의해서 인스턴스가 재부팅되지 않도록 option 을 설정할 수 있다는 답변을 받았다.