sap-oc / cookbook-openstack-network Goto Github PK
View Code? Open in Web Editor NEWThis project forked from suse-cloud/cookbook-openstack-network
Chef Cookbook - OpenStack Network
Home Page: http://openstack.org
This project forked from suse-cloud/cookbook-openstack-network
Chef Cookbook - OpenStack Network
Home Page: http://openstack.org
The neutron-ha-tool's resource script does DHCP replication on start, which should not be done, as neutron by default takes care of this through the dhcp_agents_per_network
parameter.
Make sure that release branches are used for building ocf-resource scripts from
https://github.com/openstack/openstack-resource-agents
Package for SOC6 is taking master
https://build.suse.de/package/view_file/Devel:Cloud:6/openstack-resource-agents/_service?expand=1
On our network nodes an rcopenstack-neutron-ovs-cleanup
was required prior to the reboot, to have network working, after the node starts, otherwise it hung up in crowbar_join.
@seife please provide more details from the maintenance, if you have
Discussion on neutron-ha-tool
Problems that exist within the system, that we need to live with because changing them would require major redesign.
The tool has to know the layout of the routers and agents somehow.
When we found that an agent needs to be evacuated, how do we use the above metrics to schedule router migrations?
report_interval
and agent_down_time
. report_interval
defaults to 30s
and agent_down_time
defaults to 75s
.for neutron-ha-tool's ha mode, we don't do any ssh, however this is still a requirement.
either:
We know which network nodes live close to which AZs. We also know which routers belong to which AZs, We want to use this information to move routers to network nodes that are close to compute nodes.
neutron-ha-tool can:
--l3-agent-evacuate
)--router-list-file
)--target-agent-id
)The idea is to re-use neutron-ha-tool for this job.
On 22 Apr (Saturday) a network outage happened. We suspect neutron-ha-tool moved some of the high-traffic routers to network node 1, which became unstable due to the high load. On 24 Apr some routers have been moved off network node 1, and that seemed to make the landscape stable.
HOST
parameter of --l3-agent-evacuate
refers to a host name, not a host UUID.--router-list-file
and the routers are not found, make sure that the user is notified accordingly.2017-04-24 09:28:05,093 neutron-ha-tool ERROR Failed to migrate router=61aea97d-4711-4fb3-8fa8-43b9fa4503d9 from agent=6443bcf9-18ae-456a-b340-fcc565c0cd67 to agent=b3b9971c-24c2-4b80-91b5-4d103c261a81
Traceback (most recent call last):
File "/usr/bin/neutron-ha-tool", line 627, in migrate_router_safely
wait_for_router, delete_namespace)
File "/usr/bin/neutron-ha-tool", line 671, in migrate_router
wait_router_migrated(qclient, router_id, target['host'])
File "/usr/bin/neutron-ha-tool", line 732, in wait_router_migrated
(router_id, ", ".join(remaining_ports)))
RuntimeError: Some ports are not ACTIVE on router_id=61aea97d-4711-4fb3-8fa8-43b9fa4503d9: [8820e14e-b362-4883-8e36-5b09b2e6f112]
2017-04-24 09:28:05,094 neutron-ha-tool INFO 0 routers were evacuated from L3 agent d00-25-b5-a0-03-63
2017-04-24 09:28:05,094 neutron-ha-tool ERROR 1 errors encountered during evacuation
this information is clearly not representing that this is a timeout, and that this might not be an error after all
During our maintenance in production we had routers distributed like that:
node1: 66
node2: 67
node3: 67
node4: 64
node5: 0
We triggered a neutron-ha-tool --l3-agent-rebalance and that results in:
node1: 33
node2: 67
node3: 67
node4: 64
node5: 34
We triggered it again, but the distribution did not change. The assumption was, that neutron-ha-tool would distribute routers over all l3-agent equally and not only from one.
Fix is #6
When migrating routers, ports might not become active within the time available. In this case the following stacktrace will be printed:
2017-04-24 09:28:05,093 neutron-ha-tool ERROR Failed to migrate router=61aea97d-4711-4fb3-8fa8-43b9fa4503d9 from agent=6443bcf9-18ae-456a-b340-fcc565c0cd67 to agent=b3b9971c-24c2-4b80-91b5-4d103c261a81
Traceback (most recent call last):
File "/usr/bin/neutron-ha-tool", line 627, in migrate_router_safely
wait_for_router, delete_namespace)
File "/usr/bin/neutron-ha-tool", line 671, in migrate_router
wait_router_migrated(qclient, router_id, target['host'])
File "/usr/bin/neutron-ha-tool", line 732, in wait_router_migrated
(router_id, ", ".join(remaining_ports)))
RuntimeError: Some ports are not ACTIVE on router_id=61aea97d-4711-4fb3-8fa8-43b9fa4503d9: [8820e14e-b362-4883-8e36-5b09b2e6f112]
2017-04-24 09:28:05,094 neutron-ha-tool INFO 0 routers were evacuated from L3 agent d00-25-b5-a0-03-63
2017-04-24 09:28:05,094 neutron-ha-tool ERROR 1 errors encountered during evacuation
Make it clear within the help text of neutron-ha-tool that the HOST parameter of --l3-agent-evacuate
refers to a host name, not a host UUID.
Fix is #6
If we are using a router list by saying --router-list-file
and the routers are not found, make sure that the user is notified in a sensible way, by an info message or something like that.
When we do the migration, check if the agent is still dead before migrating away any routers from it. If we find that the router is alive again, stop migrating routers away.
Fix is #6
A the moment neutron-ha-tool is part of the openstack-neutron package but this has several disadvantages:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.