dwavesystems / dwave-cloud-client Goto Github PK

Ensure Client thread pools are shutdown consistently (client.close()ed) when client is not needed anymore.

Make CWD config partially overrides user-local config, which partially overrides the system-wide config files.

For example, if /usr/local/share/dwave/dwave.conf has:

[defaults]
endpoint = <production>
client = qpu

[prod]
token = <token>

and ~/.config/dwave/dwave.conf has:

[alpha]
endpoint = <alpha>
token = <token>

Then the user should be able to run dwave ping --profile prod and dwave ping --profile alpha.

To enable total control over config loading, we should support:

• auto-detect & progressive config load (new and legacy files)
• auto-detect of only new config (skip legacy)
• explicit config file load
• no config load or auto-detect (defer to env vars and args)

https://github.com/dwavesystems/dwave_micro_client/blob/fd86de8a2a5968dab6ce0a70b771526335255494/dwave_micro_client.py#L24

Sometimes and only for OSX + Python2.7 builds, one test fails with:

.....sssssss.............................................F....ssssssssssssssssssssss...
======================================================================
FAIL: test_submit_continue_then_ok_and_error_reply (tests.test_mock_submission.MockSubmission)
Handle polling for the status of multiple problems.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/build/dwavesystems/dwave-cloud-client/tests/test_mock_submission.py", line 280, in test_submit_continue_then_ok_and_error_reply
    self._check(results1, linear, quad, 100)
AssertionError: SolverFailureError not raised
----------------------------------------------------------------------
Ran 87 tests in 0.413s
FAILED (failures=1, skipped=29)

Use case: permissive_ssl.

Currently, every non-null value is true.

It would be nice to support: on, off, true, false, 0, 1.

Something like:

$ dwave --list-system
/usr/share/dwave/dwave.conf
/usr/local/share/dwave/dwave.conf

$ dwave --list-user
/home/user/.config/dwave/dwave.conf

$ dwave --list-local
./dwave.conf

Should be "D-Wave Cloud Client"

Still needed, or can we remove it?

What's a reasonable default if multiple profiles (or only one, or even none, but with defaults section) are defined in dwave.conf?

Load the first of many (or just one)?
Load only defaults if no other sections defined?
Introduce profile key under [defaults]?

Goal: make tests' behavior predictable (don't depend on legacy .dwrc) and configurable to explicitly run on a test/integration environment or production if needed.

solver = dwave_micro_client.Connection().get_solver()

h = {v: 0 for v in solver.properties['qubits']}
future = solver.sample_ising(h, {(2, 6): -1, (0, 5): -1})
future.samples

dwave_micro_client.SolverFailureError: (300, u'Internal Error: (1, The value -20 is not of type SB-INT:INDEX.)')

given a .dwrc along these lines, note the space,

name|url, token

python3 interpreter,

conn = dwave_micro_client.Connection()
solver = conn.get_solver()
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/dwave_micro_client.py", line 309, in get_solver
    response = self.session.get(posixpath.join(self.base_url, 'solvers/remote/{}/'.format(name)))
...
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/models.py", line 440, in prepare_headers
    check_header_validity(header)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/utils.py", line 869, in check_header_validity
    raise InvalidHeader("Invalid return character or leading space in header: %s" % name)

Looks like there is an extra 'not' in the logic that is preventing dates from being passed through.

https://github.com/dwavesystems/dwave_micro_client/blob/a425ff6984f6369514359df8fdbfa52c8d5033d8/dwave_micro_client.py#L734

This can be useful when some solvers have x parameters that are not advertised yet, for example x_reinitialize_state, etc...

If user types a non-existing path to config file during dwave configure, or if the default suggestion for a new config file location is used, the CLI script fails. Reported by @orenshk.

Some unittests fail sometimes, but succeed on re-run. Maybe we have a thread-sync bug?

Canceling test fails:

$ coverage run -m unittest discover

.............................................................................E...........
======================================================================
ERROR: test_cancel_batch (tests.test_solver.Submission)
Submit batch of problems, then cancel them.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/radomir/work/dwave-cloud-client/tests/test_solver.py", line 239, in test_cancel_batch
    self.assertTrue(10000 == sum(results.occurrences))
  File "/home/radomir/work/dwave-cloud-client/dwave/cloud/computation.py", line 266, in occurrences
    self._load_result()
  File "/home/radomir/work/dwave-cloud-client/dwave/cloud/computation.py", line 308, in _load_result
    raise self.error
dwave.cloud.exceptions.SolverFailureError: Internal SAPI Error (reference # - 62816f0d-4c0b-4398-801d-526e3225cebb). Contact System Administrator

----------------------------------------------------------------------
Ran 89 tests in 188.323s

FAILED (errors=1)

To preserve comments in a .conf file on update, we should probably switch from configparser to configobj.

Run integration tests with SAPI web service during our CI build/test.

Circle CI proved to be much faster and more flexible than Travis.

.dwrc URLs are having + '/... done to them resulting in ...//... if URL ended in / already.
Should be done with URL aware appending.

Can you reproduce this on v0.2.2?

.dwrc along these lines,

name|url,token

python3 interpreter,

conn = dwave_micro_client.Connection()
solver = conn.get_solver()
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/dwave_micro_client.py", line 320, in get_solver
    self.solvers[data['id']] = Solver(self, data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/dwave_micro_client.py", line 646, in __init__
    self.id = data['id']
TypeError: list indices must be integers or slices, not str

To facilitate client library usage tracking on the server side.

Is there a way to test if a default connection is available in .dwrc before calling dwave_micro_client.Connection()? If not, this is a feature request.

As of v0.2.2 calling dwave_micro_client.Connection() without a .dwrc for me results in,

...
  File "/usr/local/lib/python2.7/site-packages/dwave_micro_client.py", line 168, in __init__
    url, token, proxies, self.default_solver = load_configuration(url)
  File "/usr/local/lib/python2.7/site-packages/dwave_micro_client.py", line 1440, in load_configuration
    raise IOError("Could not load configuration from {}".format(file_path))
IOError: Could not load configuration from /Users/carleton/.dwrc
unable to parse stdin as a json document

• make it more robust (fix things like #19)
• follow the same logic the new config has:
  • load from file, override with environment, override with arguments,
  • instead of: if minimal set of required variables is present in environment (or via arguments), use them and skip the file load
• unfortunately, the format has to stay the same
• return the first valid profile from file, don't fail if the first is invalid

(forked from #76, as this is a separate issue)

$ dwave ping
Using endpoint: https://cloud.dwavesys.com/sapi
Traceback (most recent call last):
  File "/home/bellert/git_root/circuit-fault-diagnosis-demo/env/bin/dwave", line 11, in <module>
    sys.exit(cli())
  File "/home/bellert/git_root/circuit-fault-diagnosis-demo/env/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/bellert/git_root/circuit-fault-diagnosis-demo/env/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/bellert/git_root/circuit-fault-diagnosis-demo/env/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/bellert/git_root/circuit-fault-diagnosis-demo/env/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/bellert/git_root/circuit-fault-diagnosis-demo/env/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/bellert/git_root/circuit-fault-diagnosis-demo/env/local/lib/python2.7/site-packages/dwave/cloud/cli.py", line 121, in ping
    print("Using solver: {}".format(solver.id))
UnboundLocalError: local variable 'solver' referenced before assignment

Contents of dwave.conf:

[defaults]
endpoint = https://cloud.dwavesys.com/sapi
client = qpu
token = ...

[DW_2000Q_VFYC_1]
solver = DW_2000Q_VFYC_1

[DW_2000Q_1]
solver = DW_2000Q_1

[BAY14_X_INTERNAL]
solver = BAY14_X_INTERNAL

[c4-sw_optimize]
client = cpu
solver = c4-sw_optimize

[c4-sw_sample]
solver = c4-sw_sample

from dwave.cloud.config import get_default_configfile_path
print(get_default_configfile_path())
C:\Users\jpasvolsky\AppData\Local\dwavesystem\dwave\dwave.conf

(There is an extra "\dwavesystem" between Local and dwave.)

According to dwave.cloud.config.load_config_from_file(filename=None):
On Windows, config file should be located in: C:\Users<username>\AppData\Local\dwave\client\dwave.conf

Timestamp parsing never happens due to reversed logic from bug #5, but even if it would happen, it wouldn't work due to invalid datetime.strptime call:

Should look like this:

Including, but not limited to:

• direct Client init, and following from_config-based ones
• user or local config only
• progressive system/user config
• add environment variables to the mix
• add arguments override to the mix
• explicit config path (testing use case), describe read/parse errors
• legacy config fallback (and how to turn it off)
• avoid loading config from file - use only env and arg
• examples of profile selection (with argument, with env, with config var, fallback)
• custom config vars handling (like permissive_ssl)

Include proxy examples for different profiles.

It would be nice-to-have feature to have editable (default) values when asking for user input in dwave configure.

Instead of using (several) thread pools per Client instance, share them among all clients in a thread, or even among (endpoint, token) clients in a thread.

Include standard configure usage, ping example and Bash completion instructions.

Add appveyor testing and fix as needed.

One thing that is a little odd to me is that I can setup a connection like so,

conn = dwave_micro_client.Connection(...)

with a bunch of junk data and I only notice that the connection is broken when I try to use it with a call like,

solver = conn.get_solver()

Maybe the Connection init function could make a simple test, just to make sure it can talk to the server.

The following message appears when loading a config file in Python 2:

/env/local/lib/python2.7/site-packages/dwave/cloud/config.py:114: DeprecationWarning: You passed a bytestring as `filenames`. This will not work on Python 3. Use `cp.read_file()` or switch to using Unicode strings across the board.
  files_read = config.read(filename)

Despite what the message says, the load succeeds in both Python 2 and 3.

In a variety of cases, CI being a big one, one might not have access to a SAPI connection backend. In such cases what is the best practice for testing a code that leverages the classes provided here, Connection especially?

Would it be possible to provide a fairly well developed mock Connection for such tests? Another option would be to expose some of kind open-access backend.

from dwave.system.samplers import DWaveSampler
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\jpasvolsky!git_Source\env\lib\site-packages\dwave\system\samplers_init_.py", line 1, in
from dwave.system.samplers.dwave_sampler import *
File "C:\Users\jpasvolsky!git_Source\env\lib\site-packages\dwave\system\samplers\dwave_sampler.py", line 7, in
import dwave.cloud.qpu as qpuclient
File "C:\Users\jpasvolsky!git_Source\env\lib\site-packages\dwave\cloud_init_.py", line 3, in
from dwave.cloud.client import Client
File "C:\Users\jpasvolsky!git_Source\env\lib\site-packages\dwave\cloud\client.py", line 16, in
from dwave.cloud.solver import Solver
File "C:\Users\jpasvolsky!git_Source\env\lib\site-packages\dwave\cloud\solver.py", line 7, in
from dwave.cloud.coders import encode_bqm_as_qp
File "C:\Users\jpasvolsky!git_Source\env\lib\site-packages\dwave\cloud\coders.py", line 6, in
from dwave.cloud.utils import uniform_iterator, uniform_get
File "C:\Users\jpasvolsky!git_Source\env\lib\site-packages\dwave\cloud\utils.py", line 4, in
import readline
ImportError: No module named readline

For Unix gnureadline is likely okay but if we also support Windows:

pip install gnureadline==6.3.8
Collecting gnureadline==6.3.8
Downloading gnureadline-6.3.8.tar.gz (2.5MB)
100% |################################| 2.5MB 483kB/s
Complete output from command python setup.py egg_info:
Error: this module is not meant to work on Windows (try pyreadline instead)

pip install pyreadline
Collecting pyreadline
Downloading pyreadline-2.1.zip (109kB)
100% |################################| 112kB 483kB/s
Building wheels for collected packages: pyreadline
Running setup.py bdist_wheel for pyreadline ... done
Stored in directory: C:\Users\jpasvolsky\AppData\Local\pip\Cache\wheels\9a\c7\45\fd424eb3d7875d7a61221accd593e17c7953ed5ece5ee60be9
Successfully built pyreadline
Installing collected packages: pyreadline
Successfully installed pyreadline-2.1

Then it works:

from dwave.system.samplers import DWaveSampler

https://bugs.python.org/issue18303

We should consider putting some logic in to handle this case.