Around November 13, the machine agent on one of our servers has stopped reporting to the controller. It was working in the past. There is nothing that we changed that I know of, except restarting the JVMs.
The error in the "/opt/AppDynamics/MachineAgent1/logs/machine-agent.log" is:
[AD Thread-Metric Reporter0] 14 Dec 2015 11:59:49,514 ERROR ManagedMonitorDelegate - HTTP Request failed: HTTP/1.1 500 Internal Server Error [AD Thread-Metric Reporter0] 14 Dec 2015 11:59:49,514 WARN ManagedMonitorDelegate - Error sending metric data to controller:null [AD Thread-Metric Reporter0] 14 Dec 2015 11:59:49,514 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:358) at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:115) at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:315) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:150) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:123) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:208) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:238) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:683) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:715) at java.lang.Thread.run(Thread.java:745) [AD Thread-Metric Reporter0] 14 Dec 2015 11:59:49,514 WARN ManagedMonitorDelegate - Metric Reporter Queue full. Dropping metrics. [AD Thread-Metric Reporter0] 14 Dec 2015 11:59:59,585 ERROR ManagedMonitorDelegate - HTTP Request failed: HTTP/1.1 404 Not Found [AD Thread-Metric Reporter0] 14 Dec 2015 11:59:59,585 WARN ManagedMonitorDelegate - Error sending metric data to controller:null [AD Thread-Metric Reporter0] 14 Dec 2015 11:59:59,585 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:358) at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:115) at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:315) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:150) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:123) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:208) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:238) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:683) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:715) at java.lang.Thread.run(Thread.java:745)
The command-line that starts the machine agent is:
/usr/bin/java -Xms64m -Xmx64m -jar /opt/AppDynamics/MachineAgent1/machineagent.jar
The version of the machine agent and JVM runtime are:
[main] 27 Nov 2015 16:12:08,916 INFO SystemAgent - ####################################################################################
[main] 27 Nov 2015 16:12:08,917 INFO SystemAgent - Agent Install Directory [/opt/AppDynamics/MachineAgent1]
[main] 27 Nov 2015 16:12:08,917 INFO SystemAgent - Using Agent Version [Machine Agent v3.9.6.0 GA Build Date 2014-11-20 10:15:23]
[main] 27 Nov 2015 16:12:08,933 INFO SystemAgent - JVM Runtime:
java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_64/jre
java.vm.vendor=Oracle Corporation
java.vm.name=OpenJDK 64-Bit Server VM
java.runtime.version=1.8.0_65-b17
java.io.tmpdir=/tmp
[main] 27 Nov 2015 16:12:08,933 INFO SystemAgent - OS Runtime:
os.name=Linux
os.arch=amd64
os.version=2.6.32-504.16.2.el6.x86_64
user.name=root
user.home=/root
user.dir=/
The first error after the latest machine agent restart is:
[Agent-Scheduler-1] 13 Dec 2015 00:07:02,552 ERROR RegistrationChannel - Fatal transport error: The host did not accept the connection within timeout of 5000 ms [AD Thread-Metric Reporter0] 13 Dec 2015 00:07:04,517 ERROR ManagedMonitorDelegate - Fatal transport error: The host did not accept the connection within timeout of 5000 ms [AD Thread-Metric Reporter0] 13 Dec 2015 00:07:04,517 WARN ManagedMonitorDelegate - Error sending metric data to controller:Fatal transport error: The host did not accept the connection within timeout of 5000 ms [AD Thread-Metric Reporter0] 13 Dec 2015 00:07:04,624 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Fatal transport error: The host did not accept the connection within timeout of 5000 ms at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:358) at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:115) at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:315) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:150) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:123) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:208) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:238) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:683) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:715) at java.lang.Thread.run(Thread.java:745)
There are also two App Agents (IBM) running on the same server.
Thank you for any help you can provide.