awegnergithub / stackapi Goto Github PK

Adjust LICENSE and Docs to include "and contributors" on copyright notices

I don't work for Google. I haven't ever done work for Google. This is a copy/paste error. Remove reference to Google Inc from the License.

https://github.com/AWegnerGitHub/stackapi/blob/master/LICENSE.txt#L3

This should return 1 item. Instead it returns 2:

>>> s = StackAPI('stackoverflow')
>>> s.page_size=1
>>> s.max_pages=1
>>> len(s.fetch('comments')['items'])
2

The sites end point doesn't have a maximum page size. When the StackAPI object is created, it should perform a single call to the end point instead of multiple calls at intervals of page_size

https://github.com/AWegnerGitHub/stackapi/blob/master/stackapi/stackapi.py#L73

Provide examples on how the "complex queries" work with StackAPI

http://api.stackexchange.com/docs/min-max

Hello!

Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard.

Changes to provide security against broader threats have been in place for a while, however there are still a few scenarios that can only be addressed by migrating to a separate domain.

We implemented session hijacking detection and took precautions to limit cookie usage, but there are still a number of scenarios utilizing XSS and CSRF attacks that we aren't able to protect against while hosting documentation from subdomains on the readthedocs.org domain. Moving documentation hosting to a separate domain will provide more complete isolation between the two user interfaces.

Projects will automatically be redirected, and this redirect will remain in place for the foreseeable future. Still, you should plan on updating links to your documentation after the new domain goes live.

If you notice any problems with the changes, feel free to open an issue on our issue tracker: http://github.com/rtfd/readthedocs.org/issues. If you do notice any security issues, contact us at [email protected] with more information.
Keep documenting,
Read the Docs

Update documentation to point as appropriate

StackAPIError is not imported properly

Related to the fix provided by #1

In the Advanced Usage section of the docs, several links that have anchor text to external resources are not properly linked.

http://stackapi.readthedocs.org/en/latest/user/advanced.html#send-data-via-the-api

StackAPI supports 2.7 and above. The json module is included in the standard library and will always be available.

I'm raising this as an issue not because I think there's anything wrong with StackAPI, but because I felt it would be rude to "butt in" with a pull request.

If I get consensus from the package owner, I'm happy to help implementing this.

Currently, the StackAPI class hardcodes the "base URL" for the StackExchange API to point at the "public" StackExchange (in stackapy.py):

self._base_url = 'https://api.stackexchange.com/{}/'.format(version)

However, StackExchange sells an enterprise variant (distinct from "StackOverflow for Teams") — which allows corporations to self-host an instance of StackOverflow. For such corporations, the base URL will look something like stackoverflow.myenterprise.com, and all request will need a header X-API-Key: api_key (or key query parameter).

It seems like a reasonably straightforward update to the StackAPI constructor to accept kwargs for the base URL and "authentication key" (I see there's already a class variable _api_key). The default values for these would be https://api.stackexchange.com and None, respectively. If they are set, then __init__ would set the base URL appropriately, and the fetch method could add the appropriate header.

Thoughts?

I am trying to replicate a total count query that is available in the StackExchange API, using the total filter. For example:
http://api.stackexchange.com/2.2/questions?site=gis&filter=total

Example StackAPI code is:

from stackapi import StackAPI
SITE = StackAPI('gis')
result = SITE.fetch(
    'questions',
    filter='total'
)
result

which results in a KeyError:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-73-dad27c0670d8> in <module>()
      3 result = SITE.fetch(
      4     'questions',
----> 5     filter='total'
      6 )
      7 result

/usr/local/lib/python3.6/dist-packages/stackapi/stackapi.py in fetch(self, endpoint, page, key, filter, **kwargs)
    225         r = []
    226         for d in data:
--> 227             r.extend(d['items'])
    228         result = {'backoff': backoff,
    229                   'has_more': data[-1]['has_more'],

KeyError: 'items'

I am trying to fetch all the questions tagged on a particular tag from StackExchange API. For example, I want to fetch all questions under the "swift" tag from the "StackOverflow" API.

I am using StackAPI in python but am not able to find a solution on how to fetch all questions automatically?

What I have tried:

The first solution: give a large page number so it will fetch all questions under that limit?

    from stackapi import StackAPI
    
    SITE     = StackAPI('stackoverflow')
    SITE.page_size = 100
    SITE.max_pages = 5000000    #very large max_pages number 
    posts    = SITE.fetch('questions', tagged='swift', sort='activity', order = 'desc')

That doesn't seem like an optimal solution, this second solution:

def stackapi_questions():
    SITE     = StackAPI('stackoverflow')
    posts    = SITE.fetch('questions', tagged='swift', sort='activity', order = 'desc')
    return posts
  
  
  
  def get_all_questions():
  
    has_more = True
    results  = []
    while has_more:
      response = stackapi_questions()
      results.append(response)
      has_more = response['has_more']
    return results

Here I am worried to do I need to take care of 'quota_remaining' and 'backoff' if I am using StackAPI in this way or do I have to handle those parameters' value myself?

How to fetch all questions using this API or any other better method?

I am trying to fetch notification from the site, but i cant actually stop when one notification is recieved. Like since as long as i dont see the notification, it stays un-read and all that while i keep on getting the pop notification i made. Is there anyway to fetch the notification efficiently? I dont if this is how we ask stuff here, im new to over here.

After long time i get an error too:

Exception in thread Thread-1734:
Traceback (most recent call last):
  File "C:\Users\nihaa\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\nihaa\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "c:/PyProjects/Patient Data Entry/new.py", line 39, in first
  File "C:\Users\nihaa\AppData\Local\Programs\Python\Python37-32\lib\site-packages\stackapi\stackapi.py", line 198, in fetch
    raise StackAPIError(self._previous_call, error, code, message)
stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/13382000/inbox/?pagesize=100&page=1&filter=default&key=...%28%28&access_token=...%29%29&site=stackoverflow', 502, 'throttle_violation', 'too many requests from this app/user pair, more requests available in 41391 seconds')

My code with the Q - https://stackoverflow.com/questions/63879077/displaying-desktop-notification-after-getting-a-new-message-python-tkinter

Not all end points use ids to identify a list of elements to search for. The tags end points are examples of this:

/tags/{tags}/info

Related #18

https://pypi.python.org/pypi/StackAPI

It is not nicely formatted.

The mocked objects need to be pointed to correctly, regardless of how the tests are being run.

This will require changing fabfile.py to not deploy to PyPI.

Hello!

I registered my application thus obtained app key and access token. By reading the documentation I figured out that it's possible to pass additional arguments into the fetch method, as soon as they are named the same as described in the endpoint documentation.

Sounds great. So I tried

SITE.fetch('users/{ids}/top-answer-tags', ids=user_id, access_token=access_token, key=key)

And got

stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/922184/top-answer-tags/?pagesize=1&page=1&filter=default&access_token=xxxxxxxxxxxxxxxx&site=stackoverflow', 405, 'key_required', 'keyis required whenaccess_token is passed.')

I obviously passed the key, but for some reason it doesn't appear in the url. Why is that?

If I take this url and add the key manually and test it in the browser, it works perfectly! Would be nice if this is fixed :)

I am trying to retrieve list of users ids from answers.
SITE = StackAPI('stackoverflow')
SITE.page_size = 10
SITE.max_pages = 1
answers = SITE.fetch('answers', sort ='votes')
users = [answers['items'][i]['owner']['user_id'] for i in range(len(answers['items']))]

I managed to create a list of users, but when I want to fetch them with the following line of code
data = SITE.fetch('users/{}', ids=users)
I receive the following error
stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/%7B%7D/922184;62130;95592;14113;8985;9951;92315;394487;613430;22656/?pagesize=10&page=1&filter=default&site=stackoverflow', 404, 'no_method', 'no method found with this name')

What could be the problem?

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
/opt/conda/lib/python3.8/site-packages/stackapi/stackapi.py in fetch(self, endpoint, page, key, filter, **kwargs)
    203                 response.encoding = 'utf-8-sig'
--> 204                 response = response.json()
    205             except ValueError as e:

/opt/conda/lib/python3.8/site-packages/requests/models.py in json(self, **kwargs)
    897                     pass
--> 898         return complexjson.loads(self.text, **kwargs)
    899 

/opt/conda/lib/python3.8/site-packages/simplejson/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
    524             and not use_decimal and not kw):
--> 525         return _default_decoder.decode(s)
    526     if cls is None:

/opt/conda/lib/python3.8/site-packages/simplejson/decoder.py in decode(self, s, _w, _PY3)
    369             s = str(s, self.encoding)
--> 370         obj, end = self.raw_decode(s)
    371         end = _w(s, end).end()

/opt/conda/lib/python3.8/site-packages/simplejson/decoder.py in raw_decode(self, s, idx, _w, _PY3)
    399                 idx += 3
--> 400         return self.scan_once(s, idx=_w(s, idx).end())

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

StackAPIError                             Traceback (most recent call last)
<ipython-input-4-b5a87dff5e87> in <module>
      1 from stackapi import StackAPI
----> 2 SITE = StackAPI('stackoverflow')
      3 comments = SITE.fetch('comments')

/opt/conda/lib/python3.8/site-packages/stackapi/stackapi.py in __init__(self, name, version, **kwargs)
     70 
     71         self._base_url = 'https://api.stackexchange.com/{}/' class="ansi-blue-fg">.format(version)
---> 72         sites = self.fetch('sites', filter='!*L1*AY-85YllAr2)', pagesize=1000)
     73         for s in sites['items']:
     74             if name == s['api_site_parameter']:

/opt/conda/lib/python3.8/site-packages/stackapi/stackapi.py in fetch(self, endpoint, page, key, filter, **kwargs)
    204                 response = response.json()
    205             except ValueError as e:
--> 206                 raise StackAPIError(self._previous_call, str(e), str(e), str(e))
    207 
    208             try:

StackAPIError: ('https://api.stackexchange.com/2.2/sites/?pagesize=1000&page=1&filter=%21%2AL1%2AAY-85YllAr2%29', 'Expecting value: line 1 column 1 (char 0)', 'Expecting value: line 1 column 1 (char 0)', 'Expecting value: line 1 column 1 (char 0)')

Here is the trace of the error I receive when I try to run this.

SITE = StackAPI('stackoverflow')
comments = SITE.fetch('comments')

Clueless on what it can be and how to debug!

A call like

so = StackAPI('stackoverflow', key='<my_key>')
a = so.fetch('tags/{tags}/synonyms', tags=['c#'])

throws a StackAPIError with no error message. This is due to the hash char.
From the StackOverflow API UI it works instead.

Tried encoding it with same result. Any way to go round this, or can it have a fix?

Imports are not matching README.

>>> from stackapi import StackAPI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name StackAPI

Requires

>>> from stackapi.stackapi import StackAPI

It looks like you publish to pypi with this code: https://github.com/AWegnerGitHub/stackapi/blob/602d9ce6de4b8a3e0462576365536c99a5a14c71/fabfile.py

If you could add bdist_wheel as well as sdist that would be a huge help to folk who for policy reasons would like to only install wheels.

Here are some more instructions on how to create wheels,, or you could use poetry build which is even easier once you've started using poetry.

Thanks for publishing stackapi, I've had a lot of fun hacking with it.

Please create a new tag with the fix for quota_remaining

Investigate usage of coverage.py and coveralls.io for unittest coverage.

The changes made for #3 need to be mocked. Currently, they hit the API and burn through quota.

Hi,

I'm trying to use stackapi to fetch contents for specific IDs. Below is the traceback I got:

>>> answers = SITE.fetch('/answers/{ids}/comments', ids=[1, 2, 3])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "xxx/tools/miniconda3/envs/env_bot/lib/python3.6/site-packages/stackapi/stackapi.py", line 150, in fetch
    for k, value in kwargs.items():
RuntimeError: dictionary changed size during iteration

This error is caused by stackapi/stackapi/stackapi.py line 153

150 for k, value in kwargs.items():
151     if "{" + k + "}" in endpoint:
152         endpoint = endpoint.replace("{"+k+"}", ';'.join(requests.compat.quote_plus(str(x)) for x in value))
153         kwargs.pop(k, None) # kwargs size is changed during iteration

Could you help to solve this issue? I can create a pull request if you want to.

Thanks,
CY

StackAPI currently makes the, incorrect, assumption that if you are passing something via ids, it is the last element in the end point. This is not the case in many end points.

As an example, you can pass ids to /answers/{ids}/comments.

There are also end points that accept a single id, that would fail as currently implemented. An example of this is /posts/{id}/comments/render

From the docs:

Note: StackAPI does not do any data transformation for these end point parameters. This means that fromdate needs to be passed the integer values that the API Documentation describes, not a datetime object. These are Unix timestamps/epoch values.

Proposal: Revise this functionality to allow datetime objects to be passed. StackAPI should then convert these to the appropriate timestamp values.

Reasoning: Converting to the timestamp is something the library can/should do because the API takes these values. The ids key word is already accepting a list and converting it to the appropriate format. The same reasoning should occur for datetime functionality.

Implementation One Details: Repeat functionality of the ids keyword for fromdate and todate. Will need to check if the value is a datetime object or an int.

Implementation Two Details: Check all parameters to see if the pass data is a datetime object. If it is, convert it to an int.

Comparison: Implementation one fails to account for possible datetime objects outside of the two mentioned parameters. If another end point has another datetime parameter, it will not use the same functionality.

Implementation two forces an unneeded check of all parameters. Certain parameters will never be a datetime object. Others are potentially unknown.

Replace lines like

kwargs['proxy'] if kwargs.get('proxy') else None

With just the kwargs.get()

Travis should be configured to perform unittests on commit

Currently, the max_pages defaults to 100 pages

https://github.com/AWegnerGitHub/stackapi/blob/master/stackapi/stackapi.py#L71

This means that the API his hit 100 times, by default, for high volume end points and sites. This burns through unnecessary quota. This should be adjusted much lower - 2-5? With the page_size still set at 100, this will return 200-500 items instead of the default 10,000 items.

RTD available at: http://stackapi.readthedocs.org/

Currently, to get the latest data from an endpoint, for example, questions, we need to poll it at regular intervals using the API. It would be helpful to have a function that could monitor an endpoint (provided with kwargs to limit the query to a specific use case) from time to time automatically and notify the user upon being updated. I wish SO has a streaming API like Twitter but unfortunately it doesn't. So why not make our own? Instead of having the users to add these extra lines of code for polling, make a function called monitor which will do the same.

Since it is a monitoring functionality, it would be better if it was async or at least runs on a background thread to prevent blocking of the normal commands.

By observing the parameter quota_remaining I noticed that it is always equal to quota_max.
This makes it hard to monitor the number of remaining API calls and to automate data pulling.
Would be really nice if this issue is resolved.

Set up readthedocs integration

Sections needed:

• Quick Start
• Advanced Usage
• Calling fetch with various parameters (including ids)
• Sending data to API

Basic test cases are needed to ensure basic functionality

• Check that a known site exists
• Check that an invalid site does not exist
• Check that it fails correctly when no site is provided
• Check that it correctly calls an end point that does not want an API key
• Check that it properly throws a StackAPIError exception

SITE = StackAPI('stackoverflow')
comments = SITE.fetch('comments')

Giving no results

When using kwargs in the fetch operation, if an argument is not a list but rather a plain string the
';'.join(requests.compat.quote_plus(str(x)) for x in value) joins the string characters with ;

Shouldn't the line above check for value being a list?

Example:

>> tag = 'java'
>> period = 'month'
>> site.fetch('tags/{tag}/top-answerers/{period}', tag=tag, period=period)
raise StackAPIError(self._previous_call, error, code, message)
stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/tags/j;a;v;a/top-answerers/m;o;n;t;h/?pagesize=100&page=1&filter=default&site=stackoverflow', 400, 'bad_parameter', 'period')

Hi @AWegnerGitHub, would it be possible to publish a new release of StackAPI to PyPI so others can use the changes included in #49 ?

Hi,

I am new to stackAPI. I wish to extract the body of the answers and questions. Could you please help me with it. Thanks