Issue

After logging into netscaler gateway, we received “Http/1.1 Internal Server Error 43531” error. Upon looking at the configuration, STA server in the netscaler gateway is marked as down. DDC and storefront are installed on same server. Under traffic management/lb vip was also marked as down and the storefront service group in that lb vip was down. All the probes sent to that STA/storefront server were not successful. TCP-default monitor was attached to the storerfront service group. Clicking on monitor detais option gave us “time out during tcp connection establishment stage” error.

Followed steps given in the article, NetScaler ADC - Http/1.1 Internal Server Error 43531 but it did not solve the issue.

On the netscaler shell, we have executed the command, nstcpdump.sh dst host BackendServiceIP (Ex: nstcpdump.sh dst host 10.10.10.230). Backend service is the actual server where your monitor is showing as down. It could be your DDC or storefront server. In our case, as explained above ddc and storefront are installed on same server. The tcp-default monitor attached to the service group in the lb vip was giving the error. So, we’ve used the backend service ip as the storefront server ip. In the output we can see the ack 0 value, which means netscaler is sending a packet to the backend server, but it is not receiving the reply.

On a successful reply this is how the output should look for the ack value.

Next Steps

Verify with firewall team if the traffic is blocked at firewall level. Capture wireshark data on the backend service ip server (ddc or storefront etc) and analyze the packets.

Solution

In our case, there were lot of TCP packet errors in the wireshark capture on the storefront server. We have rebooted the ddc/storefront server and service monitor started working fine. Because of this, lb vip, and the STA were up. We were able to connect to netscaler gateway without any issues.