After Configuring the AMPQ and CCO, Communication is not Established between Both Servers
All port-related nuances and requirements are provided in Phase 2: Configure Firewall Rules in the Install section.
The most common communication issue between the CCO and AMQP servers is that the RabbitMQ configuration is not completed at the appropriate time. The RabbitMQ configuration must be completed before one of the following occurrences:
Virtual Appliance Installation: Theappliance image is rebooted
Manual Installation: The core installer is rebooted on the AMQP server
You can verify if the RabbitMQ configuration was successfully completed by running the following commands and verifying the output as listed here:
Recovering and Rejoining a Cluster after a CCO Server Failure
The following process applies to any CCO server that fails. This procedure assumes the following:
The CCO3 server failed
CCO1 and CCO 2 continue to service requests for the CloudCenter platform.
A Load Balancer is configured in front of all three CCO servers
HA Proxy is running on a separate CentOS7 VM.
To address the failure of CCO3, you must build a new VM to replace the failed server.
Remove the Failed CCO IP from the CCO Load Balancer.
Edit the configuration file and reload the HA Proxy.
Reload HA Proxy to use the new configuration and verify the status to ensure that it is running.
Remove the Failed CCO IP using the HA Proxy GUI, if you have it enabled, to set the failed CCO to Maintenance Mode.
Select the Failed CCO backend and set it to MAINT mode.
Deploy a new VM to replace the Failed CCO VM. See CCO (Required) for procedural details.
Verify that the replacement CCO VM is ready. SSH (using the -i option) into the new replacement CCO VM.
The 4.8.2x appliance images block ICMP, so you may not know that the VM is up if you use the ping method. Instead use the SSH method to verify readiness.
Switch user to "root" access.
Once connected via SSH, verify the following VM settings.
Check the Interface config for Static or DHCP for your setup. Modify if you need to make changes. Save and exit.
Check the /etc/hosts file matches that of the two Running CCO VMs. If not add the entries to the Failed CCO3 Node. Save and exit.
Verify the Hostname in /etc/hostname file using the CLI command. Save and exit.
Verify DNS settings and try to resolve a search site.
Setup SSH Keys between all three CCOs. When you’re done, you should be able to SSH from each of the three CCOs to the other CCOs without being asked for a username/password.
When you boot using a Key-Pair, the .ssh directory and authorized_keys file are already there!
On one of the existing CCOs servers (in this procedural example, CCO1 or CCO2), issue the following commands.
On CCO3, copy the text key output from the above command, insert the text, save, and exit.
On CCO1 (assuming SFTP usage), validate that SSH from CCO1 is good to CCO3.
Validate the SSH from CCO2 is good to CCO3.
Validate the SSH from CCO3 is good to CCO 1 and CCO2.
Centos 7 has chmod 600 on these files and this does not need to be changed.
Error after Multiple CCOs are Restarted Simultaneously
Sometimes, CCOs do not come up successfully in HA mode due to an operation-heartbeat-timeout error.
When CCOs are simultaneously restarted, they need additional time to form a cluster and this may sometimes result in an operation-heartbeat-timeout.
Restart one CCO gateway service first and wait for it to be up completely. Then restart the other two CCO gateway services. You will know that the CCO gateway service is up completely when you see the following message in the gateway service log file (/usr/local/cliqr/logs/gateway.log):
CCOs Fails to Start with AMQP IO Exception
The following applies to 4.x CloudCenter HA environments. When HA proxy is used as a load balancer for AMQP VMs, the max-connections should be increased if the number of VMs in that region is more than 950. The max-connections is set to 2000 by default.
When HA proxy is used as a load balancer for AMQP VMs, ensure to increase the max-connections setting (default = 2000) if the number of VMs in that region exceeds 950.
You must set the max-connections to a value that is slightly more than twice the number of configured VMs.
You can configure this value in the /etc/haproxy/haproxy.cfg file.
Before changing the value be sure to save a backup of the haproxy.cfg file in another folder/location.
Add the following lines to the default haproxy.cfg file. All other lines should remain unchanged.
Add a new section called global at the start of the file and add maxconn value as displayed in the code block above.
Add a maxconn value to frontend amqps-in section as displayed in the code block above.
- No labels