Hi CloudStack Community,
We recently upgraded our CloudStack environment from version 4.19.2.0 to 4.20.2.0. The upgrade process completed successfully; however, we are encountering an issue post-upgrade.
Environment Details:
Current Cloudstack Version - 4.19.2.0
Need to Upgrade to CloudStack Version: 4.20.2.0
Hypervisor: Citrix XenServer
Zone Types: EC Zone & VPC Zone(both)
Issue Description:
After the upgrade:
Hosts in the VPC Zone are functioning correctly and appear in the UP state in the CloudStack UI. However, hosts in the EC Zone have gone into an Alert state and are not transitioning back to UP.
Observations:
Upon analyzing the logs, we observed failures related to the callHostPlugin.
After logging into the hypervisor, we found that the task host.call_plugin is stuck in a pending state.
Request:
Could you please help us identify the root cause of this issue and suggest possible steps to resolve it?
I have attached the relevant error logs and hypervisor task list output for reference.
2026-03-28 03:21:35,277 INFO [c.c.a.m.ClusteredAgentManagerImpl] (DirectAgentCronJob-4:[ctx-8fed6763]) (logid:e1cdc087) PingMap for agent: 70098 (null) will not be updated because agent is no longer in the PingMap
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServerConnectionPool] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) XmlRpcException for method: host.call_plugin due to org.apache.xmlrpc.XmlRpcException: Failed to create input stream: Read timed out
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServer650Resource] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) callHostPlugin failed for cmd: can_bridge_firewall with args instance: VM, host_uuid: 2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream: Read timed out
2026-03-28 03:21:35,280 WARN [c.c.h.x.r.w.x.CitrixSetupCommandWrapper] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Unable to setup com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: can_bridge_firewall with args instance: VM, host_uuid: 2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream: Read timed out
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:380)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.canBridgeFirewall(CitrixResourceBase.java:497)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:67)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:44)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1778)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-03-28 03:21:35,280 DEBUG [c.c.a.m.D.Task] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Seq 70097-597289900580012033: Response Received:
2026-03-28 03:21:35,281 DEBUG [c.c.a.t.Request] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Seq 70097-597289900580012033: Processing: { Ans: , MgmtId: 165385370412082, via: 70097, Ver: v1, Flags: 110, [{"com.cloud.agent.api.SetupAnswer":{"_reconnect":"true","result":"false","details":"callHostPlugin failed for cmd: can_bridge_firewall with args instance: VM, host_uuid: 2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream: Read timed out","wait":"0","bypassHostMaintenance":"false"}}] }
Cmd - xe task-list
uuid ( RO) : 77840f1d-70fb-2a31-c941-ed93c75e68be
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 2ac624c1-9584-d56e-cae7-1988fba8c0b2
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 6a9ce2ab-3353-4bc3-b23c-b17cf4eafb28
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
Please note - I have rolled back the upgrade, and it is running on CloudStack version 4.19.2.0.
Thanks in advance for your support.
Thanks
Shivam Kashyap
Hi CloudStack Community,
We recently upgraded our CloudStack environment from version 4.19.2.0 to 4.20.2.0. The upgrade process completed successfully; however, we are encountering an issue post-upgrade.
Environment Details:
Current Cloudstack Version - 4.19.2.0
Need to Upgrade to CloudStack Version: 4.20.2.0
Hypervisor: Citrix XenServer
Zone Types: EC Zone & VPC Zone(both)
Issue Description:
After the upgrade:
Hosts in the VPC Zone are functioning correctly and appear in the UP state in the CloudStack UI. However, hosts in the EC Zone have gone into an Alert state and are not transitioning back to UP.
Observations:
Upon analyzing the logs, we observed failures related to the callHostPlugin.
After logging into the hypervisor, we found that the task host.call_plugin is stuck in a pending state.
Request:
Could you please help us identify the root cause of this issue and suggest possible steps to resolve it?
I have attached the relevant error logs and hypervisor task list output for reference.
2026-03-28 03:21:35,277 INFO [c.c.a.m.ClusteredAgentManagerImpl] (DirectAgentCronJob-4:[ctx-8fed6763]) (logid:e1cdc087) PingMap for agent: 70098 (null) will not be updated because agent is no longer in the PingMap
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServerConnectionPool] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) XmlRpcException for method: host.call_plugin due to org.apache.xmlrpc.XmlRpcException: Failed to create input stream: Read timed out
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServer650Resource] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) callHostPlugin failed for cmd: can_bridge_firewall with args instance: VM, host_uuid: 2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream: Read timed out
2026-03-28 03:21:35,280 WARN [c.c.h.x.r.w.x.CitrixSetupCommandWrapper] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Unable to setup com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: can_bridge_firewall with args instance: VM, host_uuid: 2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream: Read timed out
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:380)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.canBridgeFirewall(CitrixResourceBase.java:497)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:67)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:44)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1778)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-03-28 03:21:35,280 DEBUG [c.c.a.m.D.Task] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Seq 70097-597289900580012033: Response Received:
2026-03-28 03:21:35,281 DEBUG [c.c.a.t.Request] (DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Seq 70097-597289900580012033: Processing: { Ans: , MgmtId: 165385370412082, via: 70097, Ver: v1, Flags: 110, [{"com.cloud.agent.api.SetupAnswer":{"_reconnect":"true","result":"false","details":"callHostPlugin failed for cmd: can_bridge_firewall with args instance: VM, host_uuid: 2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream: Read timed out","wait":"0","bypassHostMaintenance":"false"}}] }
Cmd - xe task-list
uuid ( RO) : 77840f1d-70fb-2a31-c941-ed93c75e68be
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 2ac624c1-9584-d56e-cae7-1988fba8c0b2
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 6a9ce2ab-3353-4bc3-b23c-b17cf4eafb28
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
Please note - I have rolled back the upgrade, and it is running on CloudStack version 4.19.2.0.
Thanks in advance for your support.
Thanks
Shivam Kashyap