Remove challenge start & end date when viewing the description for a single challenge.

Features to implement for the EvalAI CLI:

• Get all challenges

  • command: evalai challenges list
  • data: name, id and short description

• Show number of challenges in which participant has participated

  • command: evalai challenges list -participate true
  • data: Show the challenge id, challenge name, and short description

• Show number of challenges in which participant has hosted

  • command: evalai challenges list -host true
  • data: Show the challenge id, challenge name, and short description

• Get current challenges

  • command: evalai challenges list ongoing
  • data: name, id and short description

• Get past challenges

  • command: evalai challenges list past
  • data: name, id and short description

• Get future challenges

  • command: evalai challenges list future
  • data: name, id and short description

• Get details about the phases of a challenge

  • command: evalai challenges phases list -c 1
  • data: phase name, id and short description

• Get details about a particular phase of a challenge

  • command: evalai challenges phases -c 1 -p 2
  • data: show all the details that a ChallengePhase serializer shows in django app

• Get all participant teams of the user

  • command: evalai teams list
  • data: show team name, team pk, members usernames in the team

• Create a participant team

  • command: evalai teams create
  • data: show team name, team pk, member

• Participate in a challenge

  • command: evalai challenges participate -c 1 -pt 123
  • data: success message with relevant metadata

• Make submissions to the challenge phase

  • command: evalai challenges submit -c 1 -p 2 -file submission.json
  • data: show the status of the submission with all relevant information that we show in my-submissions

• Get status of a particular submission

  • command: evalai submissions -s 1234
  • data: show the status of the submission with all relevant information that we show in my-submissions

• Get all the challenge phase splits of a phase in a challenge

  • code: evalai challenges phases splits -c 1 -p 2
  • data: show the name, id of the challenge phase splits

• Pretty print the leaderboard of a challenge

  • command: evalai challenges leaderboard -c 1 -cps 3
  • data: show the leaderboard as it is shown on EvalAI website

• Show submission stats for a challenge (both for participants and hosts)

  • command: evalai challenges submissions stats -c 1
  • data: shows the stats based on the user if he is a participant or show all submission stats is he is a host

• Show submission stats for a challenge phase (both for participants and hosts)

  • commad: evalai challenges submissions stats -c 1 -p 2
  • data: shows the stats based on the user if he is a participant or show all stats is he is a host

• Display number of submissions happened between two dates for a challenge phase

  • command: evalai challenges submissions -d1 mm/dd/yyyy -d2 mm/dd/yyyy
  • data: show all the submission id, submission date and other details that we show in my-submissions tab in EvalAI

• Display number of submissions happened between two dates for a challenge phase with filtering by participant team.

  • command: evalai challenges submissions -d1 mm/dd/yyyy -d2 mm/dd/yyyy -pt 12
  • data: show all the submission id, submission date and other details that we show in my-submissions tab in EvalAI done by a particular participant team

• How to add auth token for EvalAI-CLI is not there in README
• Add contribution guidelines

I am not sure if this is necessary. But when I tried running a few commands, I saw a warning to add token.json. That's why I created this issue

Current Scenario:
Challenge statistics is visible to both challenge hosts and challenge participants.

Deliverabes:
The challenge statistics should only be visible to challenge host.

We should have templates that describe to new contributors what the expectations of the project are. This requires putting the files in a .github folder to allow Github to render them appropriately. Below is a list of files that will be needed to close this issue.

1. - CONTRIBUTING.md
2. - ISSUE_TEMPLATE.md
3. - PULL_REQUEST_TEMPLATE.md

Change the error message when token file isn’t present in specified directory.

The message should be:

The authentication token json file doesn't exists at the required path.
Please download the file from the Profile section of the EvalAI webapp and place it at ~/.evalai/token.json

Right now, we can submit a file but user should have options to fill Submission Metadata like method name, method description, project url, publication url at the time of submission.

Users should be able to create a challenge configuration zip using CLI. The idea is to have a similar setup that Sphinx provides when you want to generate the documentation configuration. It should work something like this:

Command: evalai create_challenge

$ evalai create challenge 

Enter the challenge name:
Test Challenge

Enter the Number of Phases:
2

Enter the Phase 1 name:
Test Phase

Enter the Phase 1 codename:
test-phase
.
.
.
.
.
.

CC: @isht3 @varunagrawal @RishabhJain2018

• Create a file which will have multiple tokens so that one can switch from host to participant creating some sort of a login kind of feature.
  Label - enhancement

Add support in ongoing to filter out past challenges.

Since the coverage is 72% which is quite low, hence we should add test cases in order to cover all the code, before merging the new PR's.

Deliverables: Increase the coverage to 90%.

We need to figure out a better way to save the token file name.
One way could be to have a config.json that saves these values and is loaded at startup. This will allow us to not only save custom config file names but also custom locations.
We can prompt the user at startup to enter these values if they are missing, providing the current values as defaults (similar to how ssh keys are generated).

Current Scenario: Tells the functioning of the function in a single line.

Deliverables:
We should add doc strings answering the following points:

1. Args : What argument a function accepts
2. Returns: What is the final output of the function
3. Raises: What exception it raises
4. Command (Optional): What is command used to invoke it.

The current displayed error messages are: (Notice the red background and white font color)

Instead, the same text should be displayed as

We should display a server error if the API isn’t working or is down.

Description: Revert the pull request #2 and merge again by squashing the changes into 1 merge commit with a proper commit message.

Deliverables

• Add support for copying to clipboard for all the commands listed on the docs page.
• Add an icon for copying the text.
• Show a successful message Command copied to clipboard.

@RishabhJain2018

Error message like you have not participated in the challenge is not shown when a user tries to submit to a challenge without participating in it.

The command should be evalai gettoken

The output should be the token which is set at ~/.evalai/token.json file.

Add badges in the readme.

This line will give a KeyError upon running the command:
evalai challenges 340 phases where 340 is the pk of challenge that doesn't exists.

@isht3 Can you please verify this ?

Deliverables:

• Add authentication in EvalAI Python Package
• Add below functionality to fetch challenges
• get all Challenges
• get current challenges
• get past challenges
• get future challenges

Currently, most of the request function use the same format of http request pattern except a few.

    headers = get_request_header()

    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
    except requests.exceptions.HTTPError as err:
        if (response.status_code in EVALAI_ERROR_CODES):
            validate_token(response.json())
            echo(style("Error: {}".format(response.json()["error"]), fg="red", bold=True))
        else:
            echo(err)
        sys.exit(1)
    except requests.exceptions.RequestException as err:
        echo(err)
        sys.exit(1)

This could be simplified easily by abstracting this functionality as a function.

Currently, click doesn't natively support date type. We want to incorporate this feature into the CLI to accept date from the user.

Deliverables

• Add a command evalai login <username> which will prompt for the password.
• Once the password is entered, then the command will create a auth_token for the user by using the EvalAI login API, & saves it in the token.json file at ~/.evalai/token.json in the user's system.

• Please add sample responses with each command.
• Please add options by taking the reference from here and commands can be https://docs.aws.amazon.com/cli/latest/reference/
  https://docs.docker.com/engine/reference/commandline/cli/
• Add review comments
• Can pip install evalai automatically create ~/.evalai ?

The token which is placed at ~/.evalai/token.json gets reset whenever a user runs test cases on his local machine.

Steps to reproduce:

1. Add a valid token at ~/.evalai/token.json.
2. Run the test cases using py.test.
3. Check the value of token at ~/.evalai/token.json

The token value at step 1 != value at step 3.

The command should be like evalai challenge create --file=<path_to_zip_file>
Please feel free to ask if you have doubts.

We should display an error that the token is expired and ask the user to generate again.

Right now, there are some user-specific settings that need to be configured to use the CLI with different instances of the EvalAI. Some of them are:

• Host URL.
• Token File Location
• Token File Name

The default for Host URL would be the localhost:8000, and the user could change it to their host to use different instances of EvalAI.

Bug:

Currently, when the command evalai set_token <auth_token> is copied to clipboard, then the content which is copied is evalai set_token <auth_token>.

Expected Behaviour

The copied content should be evalai set_token <auth_token>

Current Scenario

If a user wants to make submissions using evalai push command, then if docker isn't running on the users machine then the error thrown is shouwn below and it will confuse the user.

Traceback (most recent call last):
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/docker/transport/unixconn.py", line 42, in connect
    sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/docker/transport/unixconn.py", line 42, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/bin/evalai", line 10, in <module>
    sys.exit(main())
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/evalai/submissions.py", line 53, in push
    docker_image = docker_client.images.get(image)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/docker/models/images.py", line 312, in get
    return self.prepare_model(self.client.api.inspect_image(name))
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/docker/api/image.py", line 245, in inspect_image
    self._get(self._url("/images/{0}/json", image)), True
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/docker/api/client.py", line 215, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/requests/sessions.py", line 537, in get
    return self.request('GET', url, **kwargs)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
    r = adapter.send(request, **kwargs)
  File "/Users/rishabhjain/Documents/Projects/evalai-cli/env/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))

Expected Scenario

The error thrown should be Docker isn't running on your system. Please start docker before making submissions

Write docs for.

• Welcome page.
• Installation.
• Getting the token.
• Connecting CLI to the host.
• Basic commands overview.
• View Challenge, Phase, Phase Splits.
• Create a participant team.
• Participate in a challenge.
• Make a submission.
• View status of a submission.
• View all your submission.
• View the leaderboard.
• Directory Structure.
• Contributing guidelines.
• Setting theme for the doc.

By their very nature, CLI tools are harder to figure out compared to GUI tools. We will need really good docs to ensure the evalai-cli too gains traction among users.

It would be great to have automatic documentation generation from doc strings as a starting point so we can add more detailed documentation later.

Right now, line breaks are a continuous stream of hyphens, we have to change it to something aesthetically pleasing like '-'*N.

For more details please refer #29

Current Scenario.

All the challenges are displayed in which participant have participated i.e present as well as past on running the command evalai challenges --participant

Expected Behaviour.

The command evalai challenges --participant should only display the challenges in which are ongoing and the participant have participated in them.

$ evalai challenges upcoming

Doesn't work .
Instead the function that should be invoked is named future. REF.

$ evalai challenges future

@RishabhJain2018 @isht3 this is just a typo kind of issue. let me know if i can propose a quick fix. 😄

Observed Behaviour

The copyright year in the footer of docs landing page is outdated - 2018

Expected Behaviour

Make the copyright year dynamic.

@RishabhJain2018

Once we have the project in a semi-working condition, it would be nice to have automatic testing and deployment to pypi.org via travis or a similar CI/CD tool.

While displaying the future challenges, we should display the start-date instead of end-date.

We should now focus on covering up all the minor test cases in order to increase the coverage to 95%.

Make docstrings more consistent by defining it with the following terms.

• ARGS
• PARSES
• RAISES

The test cases here are to be checked & rectified.

The function get_challenge_count called when either participant or host is True first checks for host and if host is false, checks for participant.

So after activating both flags, only the host challenges are printed. We have to redesign the get_challenge_count function to account for this (and possibly rename it too).

#24