dropbox / nsot Goto Github PK

View Code? Open in Web Editor NEW

399.0 50.0 66.0 3.73 MB

Network Source of Truth is an open source IPAM and network inventory database

Home Page: https://nsot.readthedocs.io

License: Other

Python 78.66% CSS 0.90% JavaScript 8.86% HTML 11.19% Shell 0.18% Dockerfile 0.21%

python ipam network-inventory source-of-truth ssot networking network-engineers network-admin network-monitoring django

nsot's Introduction

NSoT

Network Source of Truth (NSoT) is a source of truth database and repository for tracking inventory and metadata of network entities to ease management and automation of network infrastructure.

NSoT is an API-first application that provides a REST API and a web application front-end for managing IP addresses (IPAM), network devices, and network interfaces.

Resources

Documentation
Python API client/CLI utility
IRC: #nsot on irc.freenode.net

nsot's People

Contributors

Stargazers

Watchers

Forkers

meirf jathanism gmjosack coxley tbonesteaks yashodhank dmar42 vanilladuck vincentbernat theskorm jgitau mirzawaqasahmed chepazzo maxg4 discoposse termlen0 fooelisa digideskio pkleanthous-zz nickpegg aaron1801 amb1s1 aakapoor tgupta3 dirtyonekanobi narjh27 ehsan-keshavarzian reversecipher yeled dirkakrid gsgben enterstudio yijxiang hariram32 khardsonhurley ro9ueadmin draganigajic ctmcisco paname75 elizabethlarkinnelson cclauss iagelo ryanheffernan jdhall75 rahmiy asd88645438 comerford jac494 isabella232 ngen-brett pu55yf3r tkuennen mkbt stdigitaldatacenter grumo35 tryweirder osssp forseven blanketjack cristianstanciu iq-scm bellyfat riverwolf67 iicc20

nsot's Issues

Typo in README under 'working with migrations' section.

It says "We use alembic". Fix that!

Only collect static files if SERVE_STATIC_FILES = True

If serving static files is disabled, we should also automatically disable collectstatic on server start.

Set queries on networks don't include optional filter arguments

For example --include-ips on a set query of -q hostname=foo-bar does not include any IP address objects. Perhaps we should make set queries always include IPs by default?

500 error when multiple attributes of the same name exist in >1 Site

Say you've got 2 Sites, each with a Device attribute called "owner". If you try a set query such as this:

GET /api/sites/3/devices/query?query=owner=jathan

This happens:

Internal Server Error: /api/sites/3/devices/query/
Traceback (most recent call last):
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/rest_framework/viewsets.py", line 85, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/nsot/api/views.py", line 246, in query
    objects = self.queryset.set_query(query, site_pk=site_pk)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/nsot/models.py", line 219, in set_query
    attr = Attribute.objects.get(name=name, resource_name=resource_name)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/jathan/sandbox/virtualenvs/nsot/local/lib/python2.7/site-packages/django/db/models/query.py", line 338, in get
    (self.model._meta.object_name, num)
MultipleObjectsReturned: get() returned more than one Attribute -- it returned 2!

Pull in static files at build time, removing them from our repo

See:

http://cramer.io/2014/12/19/distributing-js-with-python/

Convert all "constants" to "settings".

Anything that is "constant" should be configurable in settings. This would eliminate the need for nsot.constants, and just have everything globally default to values defined in nsot.conf.settings (which is merged into the global Django settings (from django.conf import settings), and optionally overridden by site-specific nsot.conf.py file.

Create nsot-server command to generate a new secret_key

Something like:

$ nsot-server generate_key
z76b3M0UfwsmieC6pm2QWzgNhaWyfschUFnXK9h2SWQ=

500 error when API auth_token authentication is not sent correctly

For example, a POST to /api/authenticate/ with an empty payload results in:

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 454, in dispatch
    self.response = self.finalize_response(request, response, *args, **kwargs)
  File "rest_framework/views.py", line 374, in finalize_response
    % type(response)
AssertionError: Expected a `Response`, `HttpResponse` or `HttpStreamingResponse` to be returned from the view, but received a `<type 'NoneType'>`

Looks like this related to nsot.api.views.AuthTokenLoginView

500 error when performing a set query for a non-existent attribute

This should be caught and return a 400. An unhandled ObjectDoesNotExist exception is happening. We need this to be caught and raise a ValidationError instead.

Development section of README should be more explicit about use of virtualenvwrapper

The section that calls out "if you use virtualenvwrapper" is not clear and may be confusing for new contributors. This should be improved to include a non-virtualenvwrapper step as well.

500 error when sending a null hostname on Device creation

Not that one SHOULD do this but when I send this payload:

POST /api/sites/1/devices

{"hostname": null}

I get this error:

  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/tornado/web.py", line 1332, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/Users/jathan/sandbox/src/nsot/nsot/decorators.py", line 33, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/jathan/sandbox/src/nsot/nsot/handlers/api.py", line 1005, in post
    site_id, hostname, attributes, commit=False
  File "/Users/jathan/sandbox/src/nsot/nsot/handlers/api.py", line 1026, in create_object
    hostname=hostname, attributes=attributes, commit=commit
  File "/Users/jathan/sandbox/src/nsot/nsot/models.py", line 620, in create
    obj = cls(site_id=site_id, hostname=hostname)
  File "<string>", line 4, in __init__
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 260, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line 526, in _declarative_constructor
    setattr(self, k, kwargs[k])
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 226, in __set__
    instance_dict(instance), value, None)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 693, in set
    value, old, initiator)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 712, in fire_replace_event
    state, value, previous, initiator or self._replace_token)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/util.py", line 111, in set_
    return validator(state.obj(), key, value)
  File "/Users/jathan/sandbox/src/nsot/nsot/models.py", line 642, in validate_hostname
    if len(value) < 1:
TypeError: object of type 'NoneType' has no len()

It's a simple fix, but I've got a PR in flight and don't feel like branching.

Documentation: Intro.rst

There are currently three types of permissions a User
can have in order to make modifications:

    * admin
        - Ability to Update/Delete Site
        - Ability to grant permissions within a site
        - All subsequent permissions
    * networks
        - Ability to Add/Update/Delete Networks
        - Ability to Add/Update/Delete Attributes for Networks.

Three types of permissions user can have, two described.

Unhandled 500 error citing transaction rollback issue.

Trying to hit /api/sites/1 and this happened:

2015-04-07 20:00:42,272 ERROR   Uncaught exception GET /api/sites (10.16.16.85)
HTTPServerRequest(protocol='http', host='foobar:5150', method='GET', uri='/api/sites', version='HTTP/1.1', remote_ip='10.16.16.85', headers={'Host': 'foobar:5150', 'Accept-Encoding': 'gzip, deflate, compress', 'Authorization': 'AuthToken jathan@localhost:CENSORED', 'Content-Type': 'application/json', 'Accept': 'application/json', 'User-Agent': 'python-requests/1.2.3 CPython/2.7.7 Linux/3.2.0-67-generic'})
Traceback (most recent call last):
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/tornado/web.py", line 1309, in _execute
    result = self.prepare()
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/nsot/handlers/util.py", line 192, in prepare
    BaseHandler.prepare(self)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/nsot/handlers/util.py", line 70, in prepare
    if not self.current_user:
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/tornado/web.py", line 1022, in current_user
    self._current_user = self.get_current_user()
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/nsot/handlers/util.py", line 156, in get_current_user
    user = models.User.verify_auth_token(email, auth_token)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/nsot/models.py", line 306, in verify_auth_token
    user = User.query().filter_by(email=email).scalar()
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2426, in scalar
    ret = self.one()
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2395, in one
    ret = list(self)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
    return self._execute_and_instances(context)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 893, in _execute_context
    None, None)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
    exc_info
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 887, in _execute_context
    conn = self._revalidate_connection()
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 242, in _revalidate_connection
    "Can't reconnect until invalid "
StatementError: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) u'SELECT users.id AS users_id, users.email AS users_email, users.secret_key AS users_secret_key \nFROM users \nWHERE users.email = %s' [immutabledict({})]
2015-04-07 20:00:42,274 ERROR   500 GET /api/sites (10.16.16.85) 3.77ms

Add IPAM functionality to determine next available address(es)

From a network object, something like this:

GET /api/sites/1/network/15/next_address?num=2
{
    "addresses": [
        "10.15.23.1/32",
        "10.15.23.2/32"
    ]
}

REST Hooks

@jathanism and I talked in #nsot and dropbox/pynsot#55 about this.

This is really important to have for me and brings Trigger + source of truth integration so much closer (better, at least).

Endless possibilities here, but for a Trigger use-case keeping a NetDevices loader up to date would be easy and fast. Could let Trigger do what it does best at loading via JSON, YAML, Mongo, SQLite, etc and just run a small service to catch the updates from NSoT.

References:

Need to decide on what events we want to create. Obvious ones (I think) would be:

Site
Attribute
Network
Device

Unable to create sites when running dev server

When attempting to create the first site using the dev server, the following error is presented:

403 - '_xsrf' argument missing from POST

The server does not seem to be setting the cookie from which the XSRF token is expected to be extracted.

Crash when "restrict_networks" is null in config.yaml

When restrict_networks is null in config.yaml, the following happens:

$ tail nsot.log
    main(sys.argv)
  File "/srv/nsot/envs/nsot/bin/nsot-server", line 37, in main
    settings.update_from_config(args.config)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/nsot/settings.py", line 33, in update_from_config
    value = override(value)
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/nsot/settings.py", line 56, in override_restrict_networks
    new_values.append(ip_network(value))
  File "/srv/nsot/envs/nsot/lib/python2.7/site-packages/ipaddress.py", line 148, in ip_network
    address)
ValueError: u'None' does not appear to be an IPv4 or IPv6 network

We should allow a null value to equate to the defaults, perhaps?

Implement support for -v/--verbosity flag in nsot-server

Now that the Django transition is done, we need to implement support for toggling verbosity within Django's built-in logger.

Example here:
https://gist.github.com/epicserve/7236266

Add ability to do rollback of objects

Since we have the full object stored in the changes table. Do we want to add the ability to easily "rollback" a change?

DoesNotExist when an attribute name isn't found when performing a set query

Python stack trace:

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "rest_framework/viewsets.py", line 85, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "nsot/api/views.py", line 246, in query
    objects = self.queryset.set_query(query, site_pk=site_pk)
  File "nsot/models.py", line 219, in set_query
    attr = Attribute.objects.get(name=name, resource_name=resource_name)
  File "django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "django/db/models/query.py", line 334, in get
    self.model._meta.object_name
DoesNotExist: Attribute matching query does not exist.

The query that resulted in this crash was:

GET /api/sites/1/devices/query/query=role=[pr, dr, br]

Change "include_ips" to the default to True when retrieving Networks

More often than not when searching for IPs using the API, I am thrown off because the IP addresses (where is_ip=True on the object) are not shown by default.

I propose we flip the default behavior of include_ips search filter to True.

When database goes away, client connections just hang.

For example, the MySQL server became unreachable and/or the hostname no longer resolves:

2015-04-11 15:03:29,889 ERROR   Uncaught exception POST /api/authenticate (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:8990', method='POST', uri='/api/authenticate', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Content-Length': '91', 'Host': 'localhost:8990', 'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/json', 'Connection': 'keep-alive', 'Accept': '*/*', 'User-Agent': 'python-requests/2.5.1 CPython/2.7.8 Darwin/14.3.0'})
Traceback (most recent call last):
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/tornado/web.py", line 1309, in _execute
    result = self.prepare()
  File "/Users/jathan/sandbox/src/nsot/nsot/handlers/util.py", line 203, in prepare
    BaseHandler.prepare(self)
  File "/Users/jathan/sandbox/src/nsot/nsot/handlers/util.py", line 81, in prepare
    if not self.current_user:
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/tornado/web.py", line 1022, in current_user
    self._current_user = self.get_current_user()
  File "/Users/jathan/sandbox/src/nsot/nsot/handlers/api.py", line 2561, in get_current_user
    user = self.session.query(models.User).filter_by(email=email).scalar()
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2426, in scalar
    ret = self.one()
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2395, in one
    ret = list(self)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
    return self._execute_and_instances(context)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2451, in _execute_and_instances
    close_with_result=True)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2442, in _connection_from_session
    **kw)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 854, in connection
    close_with_result=close_with_result)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 858, in _connection_for_bind
    return self.transaction._connection_for_bind(engine)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 322, in _connection_for_bind
    conn = bind.contextual_connect()
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1798, in contextual_connect
    self.pool.connect(),
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/pool.py", line 338, in connect
    return _ConnectionFairy._checkout(self)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/pool.py", line 644, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/pool.py", line 442, in checkout
    dbapi_connection = rec.get_connection()
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/pool.py", line 526, in get_connection
    self.connection = self.__connect()
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/pool.py", line 538, in __connect
    connection = self.__pool._creator()
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 96, in connect
    connection_invalidated=invalidated
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 90, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 377, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (OperationalError) (2005, "Unknown MySQL server host 'ubuntu.local' (0)") None None
2015-04-11 15:03:29,909 ERROR   Exception closing connection <_mysql.connection closed at 7fdd6c1aae20>
Traceback (most recent call last):
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/pool.py", line 250, in _close_connection
    self._dialect.do_close(connection)
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 412, in do_close
    dbapi_connection.close()
ProgrammingError: closing a closed connection

And the client sees this and just hangs indefinitely:

$ DEBUG=1 ./nsot networks update -i 5 -a "vlan=300"
DEBUG:pynsot.dotfile:Enforcing permissions 600 on /Users/jathan/.pynsotrc
DEBUG:rcfile:files read: [u'/Users/jathan/.pynsotrc', '/Users/jathan/.pynsotrc']
DEBUG:pynsot.commands.callbacks:TRANSFORM_ATTRIBUTES [IN]: (u'vlan=300',)
DEBUG:pynsot.commands.callbacks: name = u'vlan'
DEBUG:pynsot.commands.callbacks:value = '300'
DEBUG:pynsot.commands.callbacks:TRANSFORM_ATTRIBUTES [OUT]: {u'vlan': '300'}
DEBUG:pynsot.client:Getting token for user data: {'secret_key': 'XXXXXXXX', 'email': 'jathan@localhost'}
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost

Implement object-level permissions.

This isn't yet a requirement, but it will be. And when it becomes one, django-guardian is likely going to be what we'll use:

http://django-guardian.readthedocs.org/

nsot-server request log should show X-Forwarded-For IP

Currently in only shows REMOTE_ADDR and that breaks down when you're behind nginx, for example.

API sometimes returns 409 Conflict Forbidden when trying to DELETE a resource

I can't quite figure out why this is happening. So here we go...

Using pynsot client I am trying to delete this object:

{
    "is_ip": false,
    "site_id": 1,
    "network_address": "10.16.1.0",
    "parent_id": 4,
    "prefix_length": 25,
    "ip_version": "4",
    "attributes": {
        "ospf_area": "1.1.17.255"
    },
    "id": 218
}

Like so:

In [111]: api.sites(1).networks(218).delete()
---------------------------------------------------------------------------
HttpClientError                           Traceback (most recent call last)
<ipython-input-111-039f231ebcaf> in <module>()
----> 1 api.sites(1).networks(218).delete()

/home/jathan/sandbox/virtualenvs/pynsot/lib/python2.7/site-packages/slumber/__init__.pyc in delete(self, **kwargs)
    163
    164     def delete(self, **kwargs):
--> 165         resp = self._request("DELETE", params=kwargs)
    166         if 200 <= resp.status_code <= 299:
    167             if resp.status_code == 204:

/home/jathan/sandbox/virtualenvs/pynsot/lib/python2.7/site-packages/slumber/__init__.pyc in _request(self, method, data, files, params)
     99
    100         if 400 <= resp.status_code <= 499:
--> 101             raise exceptions.HttpClientError("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
    102         elif 500 <= resp.status_code <= 599:
    103             raise exceptions.HttpServerError("Server Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)

HttpClientError: Client Error 409: http://localhost:5050/api/sites/1/networks/218

And the nsot server log shows this:

2015-04-02 00:36:39,745 DEBUG   token_auth authenticated user: jathan@localhost
2015-04-02 00:36:39,769 WARNING 409 DELETE /api/sites/1/networks/218 (127.0.0.1): foreign key constraint failed
2015-04-02 00:36:39,770 WARNING 409 DELETE /api/sites/1/networks/218 (127.0.0.1) 29.67ms

Based on this it's not clear at all why this is happening. I also checked to make sure that there aren't any IP addresses inside of this network 10.16.1.0/25:

In [112]: all_networks = api.sites(1).networks.get(include_ips=True)
In [113]: networks = all_networks['data']['networks']
In [114]: [n for n in networks if '10.16.1.' in n['network_address']]
Out[114]:
[{u'attributes': {u'ospf_area': u'1.1.17.255'},
  u'id': 218,
  u'ip_version': u'4',
  u'is_ip': False,
  u'network_address': u'10.16.1.0',
  u'parent_id': 4,
  u'prefix_length': 25,
  u'site_id': 1}]

/cry

Changes always links to first site

No matter which site you select (such as if you have multiple sites e.g. /sites/1, /sites/2), the link for "Changes" is always "/sites/1/changes"

Add a snippet to the web UI to show the version.

Self-explanatory!

Add support for querying Devices by hostname

e.g.

GET /api/sites/1/devices?hostname=foo-bar1

Detail view in web UI for a Device should show all current attributes/values

Currently you can only see the values if you click "Update Device". Let's make it more visible. :)

Development section of README should be more explicit about use of git clone

It currently says to use git clone [email protected]:dropbox/nsot.git, which will fail if someone is not familiar w/ GitHub or has not setup an SSH key on their account, or their host does not have their public key.

There should be a callout to setup SSH keys w/ GitHub, or just have the docs point to the https URL instead to avoid this potential issue.

Linked Attributes

Would be convenient to have "linked attributes" so that one is only required if another is provided. Useful example (using Trigger):

If OOBTerminalServerFQDN, then must also specify OOBTerminalServerPort

Bug in Interface when providing MAC address as an integer

The serializer for the API endpoint for Interfaces is casting incoming integer representations of MAC addresses as strings, thereby storing them incorrectly.

For example:

# This should be '00:1c:73:2a:60:62'
>>> import netaddr
>>> mac = 122191241314
>>> print netaddr.EUI(mac, dialect=netaddr.mac_unix_expanded)
00:1c:73:2a:60:62

# This should NOT be '12:21:91:24:13:14'
>>> print netaddr.EUI(str(mac), dialect=netaddr.mac_unix_expanded)
12:21:91:24:13:14

I've confirmed that the mac_address field for nsot.api.serializers.InterfaceSerializer is a CharField and needs to be corrected.

Bug: nsot-ctl cannot be run without mrproxy

Mrproxy is not defined as a core dependency, and nsot-ctl crashes when trying to run the migrations on a new NSoT installation:

$ /srv/nsot/envs/nsot/bin/python  /srv/nsot/envs/nsot/bin/nsot-ctl -c /etc/nsot/config.yaml migrations latest
Traceback (most recent call last):
  File "/srv/nsot/envs/nsot/bin/nsot-ctl", line 12, in <module>
    from mrproxy import UserProxyHandler
ImportError: No module named mrproxy

Implement default values for Attributes

Somewhat similar mindset as #84, having attribute defaults is pretty important for required attributes. May have a handful and attributes that are required, but may have a sane default that should be used before yelling at the user.

Network objects should accept network_address, prefix_length fields on create

Currently you can only provide cidr. Similarly to the database model object, if cidr is provided it should supersede network_address, prefix_length, otherwise if cidr is not provided, the other 2 are required.

Catch CLI errors from underlying commands and make them pretty.

This:

$ nsot-server start
Performing upgrade before service startup...
Performing collectstatic before service startup...
Traceback (most recent call last):
  File "/tmp/nsot/envs/nsot/bin/nsot-server", line 8, in <module>
    load_entry_point('nsot==0.8.5', 'console_scripts', 'nsot-server')()
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/nsot/util.py", line 192, in main
    initializer=initialize_app,
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/logan/runner.py", line 193, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/core/management/base.py", line 393, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/raven/contrib/django/management/__init__.py", line 41, in new_execute
    return original_func(self, *args, **kwargs)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
    output = self.handle(*args, **options)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/nsot/management/commands/start.py", line 77, in handle
    call_command('collectstatic', interactive=False, ignore=['src'])
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/raven/contrib/django/management/__init__.py", line 41, in new_execute
    return original_func(self, *args, **kwargs)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
    output = self.handle(*args, **options)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle
    collected = self.collect()
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 107, in collect
    handler(path, prefixed_path, storage)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 305, in copy_file
    if not self.delete_file(path, prefixed_path, source_storage):
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 255, in delete_file
    self.storage.delete(prefixed_path)
  File "/tmp/nsot/envs/nsot/lib/python2.7/site-packages/django/core/files/storage.py", line 289, in delete
    os.remove(name)
OSError: [Errno 13] Permission denied: '/etc/nsot/staticfiles/build/js/app.min.js'

Should print out pretty instead.

Read auth header from settings vs. hard-coded inside of user_proxy command

Currently it's hard coded. Let's let it read from settings.py by default, or even allow it to be overridden w/ a flag at runtime.

Devices need better validation for hostnames

For example, this should not be possible:

500 error when sending sending a null cidr on Network creation

When I do this:

POST /api/sites/1/networks

{"cidr": null}

This happens:

Traceback (most recent call last):
  File "/Users/jathan/sandbox/virtualenvs/nsot/lib/python2.7/site-packages/tornado/web.py", line 1332, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/Users/jathan/sandbox/src/nsot/nsot/decorators.py", line 33, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/jathan/sandbox/src/nsot/nsot/handlers/api.py", line 1465, in post
    site_id, cidr, attributes, commit=False
  File "/Users/jathan/sandbox/src/nsot/nsot/handlers/api.py", line 1486, in create_object
    cidr=cidr, attributes=attributes,
  File "/Users/jathan/sandbox/src/nsot/nsot/models.py", line 828, in create
    if network.network_address == network.broadcast_address:
AttributeError: 'NoneType' object has no attribute 'network_address'
2015-03-06 11:58:37,003 ERROR   500 POST /api/sites/1/networks (127.0.0.1) 8.87ms

Make network objects round-trippable from API

Right now to create a network object you send this:

{
    "attributes": {
        "hostname": "foo-bar1"
    },
    "cidr": "192.168.0.0/25"
}

But the API returns it like this:

{
    "is_ip": false,
    "site_id": 1,
    "network_address": "192.168.0.0",
    "parent_id": 6,
    "prefix_length": 25,
    "ip_version": "4",
    "attributes": {
        "hostname": "foo-bar1"
    },
    "id": 7
}

We should make it where an object that comes out from the API could be used to create the object, too.

Add IPAM functionality to determine next available network(s)

From a network object, somthign like this:

GET /api/sites/1/network/15/next_network?cidr=28
{
    "networks": [
        "10.15.23.16/28"
    ]
}

Add support for querying Networks by CIDR/network_address/prefix_len

e.g.

GET /api/sites/1/networks?network_address=10.0.0.0&prefix_len=8

or maybe even

GET /api/sites/1/networks?cidr=10.0.0.0/8

Support for regex-based attribute lookups via the API

While set unions allow for an enumerated query of multiple values of the same attribute key, it would be useful if a regex could be passed via the client api as well:

set union-based query (current implementation):

nsot.devices.query.get(query='facility=iad1 +facility=iad3 +facility=iad4 role=pr +role=br +role=dr')

vs. regex-based query (proposed):

nsot.devices.query.get(query='facility_regex=iad[134] role_regex=[pbd]r')

Make sure that all model and serializer fields are properly documented.

This includes help_text, verbose_name, label, default values, etc for every field, so that this cascades to serializer/form fields. In the case of custom model/serializer fields, make sure that the serializer fields are inheriting these values from the model field of the same name.

This is important, because this data will be populated by the Browsable API (when viewing /api in a desktop browser), and the auto-generated Swagger docs (/docs).

/favicon.ico redirects to /favicon.ico/

Small thing but we probably shouldn't append slashes to favicon when serving static files from django.

API returns 403 Forbidden when trying to DELETE a non-existent resource

For example when trying to DELETE a Site object that doesn't exist:

2015-03-04 09:40:59,561 WARNING 403 DELETE /api/sites/5 (127.0.0.1): Lacking appropriate permissions.
2015-03-04 09:40:59,561 WARNING 403 DELETE /api/sites/5 (127.0.0.1) 2.35ms

Longer term, it's probably more correct to return a 404, since that's what all the other objects are returning.

Skin the browsable API to look like the web UI

Self-explanatory.

See: http://www.django-rest-framework.org/topics/browsable-api/#customizing

Sometimes 409 conflict happens when trying to DELETE an object

From the log:

2015-04-24 17:41:42,362 WARNING 409 DELETE /api/sites/1/devices/2683 (10.16.16.21) 20.15ms

And for a little context, some debug output from nsot CLI command:

$ DEBUG=1 nsot --verbose devices remove -i 2683
DEBUG:pynsot.client:Reading dotfile.
DEBUG:pynsot.dotfile:Enforcing permissions 600 on /home/jathan/.pynsotrc
DEBUG:rcfile:files read: [u'/home/jathan/.pynsotrc', '/home/jathan/.pynsotrc']
DEBUG:pynsot.client:Validating auth_method: auth_token
DEBUG:pynsot.client:Getting token for user data: {'secret_key': 'XXXXXXXX', 'email': 'jathan@localhost'}
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
DEBUG:requests.packages.urllib3.connectionpool:"POST /api/authenticate HTTP/1.1" 200 164
DEBUG:pynsot.client:Got response: <Response [200]>
DEBUG:pynsot.commands.callbacks:GOT DEFAULT_SITE: 1
DEBUG:pynsot.commands.callbacks:GOT PROVIDED SITE_ID: None
DEBUG:pynsot.app:removing 2683
DEBUG:pynsot.app:Got site_id: 1
DEBUG:pynsot.app:Site_id found; rebasing API URL!
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
DEBUG:requests.packages.urllib3.connectionpool:"DELETE /api/sites/1/devices/2683 HTTP/1.1" 409 58
DEBUG:pynsot.app:API ERROR: {u'status': u'error', u'error': {u'message': u'', u'code': 409}}
DEBUG:pynsot.app:FORMATTING MESSAGE: u''
DEBUG:pynsot.app:ERROR MESSAGE = u''
[FAILURE] 409 CONFLICT
409 CONFLICT trying to remove device with args: id=2683

Constraints are not documented in POST or PUT for Attributes

IP addresses are not displayed when filtering Networks via attributes and include_ips

Tagging 3 networks (192.168.0.0/24, 192.168.0.0/25, 192.168.0.1/32) with attribute hostname=foo-bar1 and retrieving them via the API, returns this:

GET /api/sites/1/networks?include_ips=True&attributes=hostname%3Dfoo-bar1
{
    "status": "ok",
    "data": {
        "total": 2,
        "limit": null,
        "networks": [
            {
                "is_ip": false,
                "site_id": 1,
                "network_address": "192.168.0.0",
                "parent_id": 1,
                "prefix_length": 24,
                "ip_version": "4",
                "attributes": {
                    "hostname": "foo-bar1"
                },
                "id": 6
            },
            {
                "is_ip": false,
                "site_id": 1,
                "network_address": "192.168.0.0",
                "parent_id": 6,
                "prefix_length": 25,
                "ip_version": "4",
                "attributes": {
                    "hostname": "foo-bar1"
                },
                "id": 7
            }
        ],
        "offset": 0
    }
}

Note that include_ips=True should also include the IP 192.168.0.1/32, but the it's not there. ¡No me gusta!