brandwatchltd / bcr-api Goto Github PK

When trying to instantiate BWProject with username and project i receive:

13:50:59 DEBUG: https://api.brandwatch.com/projects

{"error":"unauthorized","error_description":"Invalid authentication credentials found on request"}

I have a bearer token in tokens.txt and I've tried the same bearer token manually via requests on my own environment and can access the project and queries so the token works. Might be an issue with the version of requests?
I'm currently using:

python - 3.7.6
requests - 2.22.0

Whether I authenticate in terminal with bcr-authenticate or with BWUser (both successful), I get an error with BWProject assigning either project name or ID to YOUR_PROJECT:

YOUR_ACCOUNT = "{name}@{domain}.com"
YOUR_PROJECT = {projectID}

project = BWProject(username=YOUR_ACCOUNT, project=YOUR_PROJECT)
17:26:04 DEBUG: https://api.brandwatch.com/projects

For the sorry souls who follow: This isn't a problem - if you get this debug message, you can just keep working normally!

Dear manager,
i'm trying to connect "bcr-authenticate" in command line
but i got a message below

-----below----
Authenticating user: [email protected]
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\util\connection.py", line 84, in create_connection
raise err
File "c:\programdata\anaconda3\lib\site-packages\urllib3\util\connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 978, in _validate_conn
conn.connect()
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connection.py", line 309, in connect
conn = self._new_conn()
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x000002BF7CAD04C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\util\retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.brandwatch.com', port=443): Max retries exceeded with url: /oauth/token?username=myemail%40email.com&grant_type=api-password&client_id=brandwatch-api-client (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002BF7CAD04C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\programdata\anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\Scripts\bcr-authenticate.exe_main.py", line 7, in
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\authenticate.py", line 66, in main
user = authenticate(args.username, args.password, credentials_path=args.store)
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\authenticate.py", line 22, in authenticate
return BWUser(username=username, password=password, token_path=credentials_path)
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\bwproject.py", line 60, in init
self.username, self.token = self._get_auth(
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\bwproject.py", line 94, in _get_auth
token = requests.post(
File "c:\programdata\anaconda3\lib\site-packages\requests\api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.brandwatch.com', port=443): Max retries exceeded with url: /oauth/token?username=myemail%40email.com&grant_type=api-password&client_id=brandwatch-api-client (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002BF7CAD04C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

------end

I took action solving this error

1. Got permission "app.brandwatch.com' Firewall by company security team. (Address: 104.199.59.157)
2. tried to get a token, using Authentication method by Brandwatch provide

'curl -X POST --data-urlencode 'password=[mypassword]' \ 'https://api.brandwatch.com/oauth/token?username=[[email protected]]&grant_type=api-password&client_id=brandwatch-api-client'

but all ways failed. please help to solve this problem.
thank you

(installed python version : Python 3.8.5)

The fix in this commit 7aba3eb is not in the pypi package of 1.3.1, thus the rule validation function is broken.
Can someone please republish it to pypi?

Some people have raised that they don't know the date format expected by methods such as get_chart(). We could add this as a note in the docstring

Hi,

I'm curious if the search feature that you can apply additional queries to backfilled queries on Web UI is also available through API calls.

Thanks,
Q

11:43:52 INFO: Mentions page 30 of queries Himelboim-COVID-19 Vaccine retrieved
11:43:53 DEBUG: https://api.brandwatch.com/projects/1998295055/data/mentions/fulltext?queryId=2000452202&startDate=2021-01-17T05%3A00%3A00&endDate=2021-02-16T05%3A00%3A00&pageSize=5000&cursor=AQ%3DAA%3DAXd32J5I%3DMjAwMDQ1MjIwMiFmN2VmMGMxNTE4ZGIzMzQ1ZDE4MmIzNTRkMzRkM2JiZg

KeyError Traceback (most recent call last)
in
----> 1 filtered = queries.get_mentions(name = "Himelboim-COVID-19 Vaccine ",
2 startDate = start,
3 endDate = today)

~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bcr_api\bwdata.py in get_mentions(self, name, startDate, max_pages, **kwargs)
34 key: value for (key, value) in kwargs.items() if key != "iter_by_page"
35 }
---> 36 all_mentions = list(
37 self.iter_mentions(
38 name=name,

~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bcr_api\bwdata.py in iter_mentions(self, name, startDate, max_pages, iter_by_page, **kwargs)
78 else:
79 page_idx += 1
---> 80 next_cursor, next_mentions = self._get_mentions_page(params)
81 if len(next_mentions) > 0:
82 cursor = next_cursor

~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bcr_api\bwdata.py in _get_mentions_page(self, params)
994 raise KeyError("Mentions GET request failed", mentions)
995
--> 996 return mentions.get("nextCursor", None), mentions["results"]
997
998 def _valid_input(self, param, setting):

KeyError: 'results'

Solved

@bw-matthew noticed that bwproject.py seems to override the logging level to:
logger.setLevel(logging.DEBUG). This doesn't seem right

Hi,

Can I choose "Legacy Instagram" in contentSources during using BWQueries(project).upload as it can be chose as an option of contentsources on brandwatch website UI?

Does BWSignals(project) send notification emails to subscribers when events like backfills are completed with notificationThreshold =1 ?

Is there any api feature that checks status of backfill because I would like to track how long some backfill takes ?

thanks,
Q

Maybe the field is removed in BCR? The upload method seems to work after commenting out this line:

Hi,

I’ve been using Brandwatch actively nowadays for a project almost like requesting hundreds of backfills and extracting data every day.
However, I’m stuck while I was trying Brandwatch API calls to extract data more efficiently and requesting backfills.

First, can I choose "Legacy Instagram" in contentSources during using BWQueries(project).upload as it’s available on Web UI ?
contentSources = ["blog","forum","news","twitter","review", "instagram","facebook","qq","reddit","tumblr","youtube","lexis_nexis_licensed_news"]

Does BWSignals(project) send notification emails to subscribers when events like backfills are completed if I set notificationThreshold = 1 ?
signals.upload(name= "signal_test",
queries= ["sample_query"],
subscribers= [{"emailAddress": [email protected], "notificationThreshold": 1}])

Is there any API feature that checks status of backfill or get backfill time because I would like to track how long some backfill takes ?

Also, I found out that data extracted through UI which is dashboard and data extracted through APIs are slightly different. ( difference could be more than thousands in terms of mention volume )
I assume that timezone or time setting has caused the data difference when using API BWQueries.get_chart.
I guess if I set Date on Brandwatch Dashboards as from 2019-01-01 to 2021-01-31, it automatically applies as the project region timezone, which is Asia/Seoul that I set before and the date would be converted as internally like 2019-01-01 +0900 and 2021-01-31 +0900 and monthly sum data is calculated starting from every 01-01 +0900 not 01-01 +0000. causes the data difference
And I understand startDAte and endDate are not for setting the first date of each month when providing monthly data but
I tried queries.get_chart in two different ways like below
queries.get_chart(name=i,
startDate = "2019-01-01",
endDate = "2021-01-31",
y_axis= "volume",
x_axis= "months",
breakdown_by = "queries"
)
I also tried to set startDate and endDate as below to get date based on Asia/Seoul(KST) standard
queries.get_chart(name=i,
startDate = "2019-01-01T00:00:00.000+0900",
endDate = "2022-02-01T00:00:00.000+0900",
y_axis= "volume",
x_axis= "months",
breakdown_by = "queries"
)
Even though default timezone of my project in Brandwatch is already set as Asia/Seoul(KST), the results of API call “BWQueries.get_chart” below, which is for getting monthly data, says that the first date of each monthly data ( results part ) is from UTC 0 not UTC +9 which is Asia/Seoul(KST) standard
{'dimension1': 'months',
'dimension2': 'queries',
'aggregate': 'volume',
'results': [{'id': '2019-01-01T00:00:00.000+0000',
'name': '2019-01-01 00:00:00.0',
'data': {'timeUnit': 'months'},
'values': [{'id': 2000877612,
'name': 'C brand',
'data': {},
'value': 5}]},

Could you share how to set a first date of each month as Asia/Seoul(KST) standard for monthly data using API BWQueries.get_chart ?

Please let me know if you are available for a quick chat over Zoom
Thanks,
Q

When iterating through BWQueries.iter_mentions() there's no way to keep track of current page, retry or skip a page that yields an error. What's more, internal server errors remain uncaught - there's error key in the response, but the code looks for errors: https://github.com/BrandwatchLtd/bcr-api/blob/master/src/bcr_api/bwdata.py#L993

I have been trying for long to download via api or in the app at least 7 days but all the methods are getting wrong data. Is there any way i can get the snippet via api?

After Downloading the data ,I could see that the twitter data is empty under fullText ,I cant see any data under that domain twitter in the dump downloaded

If I do queries.upload() with samplePercentage=10, I would expect to see the samplePercentage field when I .get() that query to be set to 10. However, it is actually set to 100. @ctafoyabwatch did some investigation, and it looks like the userRequestedSampling field needs to be passed in as True for a user to manually set the sample rate (as opposed to BCR deciding the sample rate).

A simple fix might be to use to see if a user has passed in samplePercentage when uploading a query, and if so, set userRequestedSampling to True. We'll need to make sure this overrides the default value of 100.

Hello,

We are using this package to create a query over brandwatch. Previously we were able to do that with the same code base but right now it seem to be failing with the following error message, not sure if anything have been changed from Brandwatch side and if this API needs to be updated.

Tried the same code with 1.4.0, 1.5.1 version of bcr-api

Code:

project = BWProject(username=YOUR_ACCOUNT, project=YOUR_PROJECT, token_path=YOUR_TOKEN_PATH)
queries = BWQueries(project)

corporation_booleanQuery = '("Sasol" OR "Rio Tinto" OR "DRDGold" OR "Durban Roodepoort Deep Limited") AND (site:cer.org.za)'

queries.upload(name = 'testing 0' ,
             booleanQuery = corporation_booleanQuery,
             startDate = '2019-01-01',
             languages = ['en'],
             contentSources = ['news'])

Error message:

05:22:23 ERROR: There was an error with this request: 
https://api.brandwatch.com/query-validation
{"booleanQuery": "(\"Sasol\" OR \"Rio Tinto\" OR \"DRDGold\" OR \"Durban Roodepoort Deep Limited\") AND (site:cer.org.za)", "language": ["en"]}
[{'code': 0, 'message': "Unrecognized field 'language'"}]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/var/tmp/ipykernel_6175/1702768516.py in <module>
     17              startDate = query_start_date,
     18              # languages = languages,
---> 19              contentSources = ['news'])
     20     corporation_queries.append(f'{backfill_portfolio_name} Client Onboarding Mentions ' + str(idx)) # entities addition
     21     # corporation_queries.append(f'Corporation Mentions ' + str(idx))  # domains addition

/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in upload(self, create_only, modify_only, **kwargs)
    296             The uploaded query information in a dictionary of the form {query1name: query1id}
    297         """
--> 298         return self.upload_all([kwargs], create_only, modify_only)
    299 
    300     def upload_all(self, data_list, create_only=False, modify_only=False):

/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in upload_all(self, data_list, create_only, modify_only)
    320 
    321         queries = super(BWQueries, self).upload_all(
--> 322             data_list, create_only=False, modify_only=False
    323         )
    324 

/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in upload_all(self, data_list, create_only, modify_only)
    162         for data in data_list:
    163             # eventually make _fill_data() a BWResource func
--> 164             filled_data = self._fill_data(data)
    165             name = data["name"]
    166 

/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in _fill_data(self, data)
    506         # validating the query search - comment this out to skip validation
    507         self.project.validate_query_search(
--> 508             booleanQuery=filled["booleanQuery"], language=["en"]
    509         )
    510         return json.dumps(filled)

/opt/conda/lib/python3.7/site-packages/bcr_api/bwproject.py in validate_query_search(self, **kwargs)
    138 
    139         valid_search = self.request(
--> 140             verb=requests.post, address="query-validation", data=json.dumps(kwargs)
    141         )
    142         return valid_search

/opt/conda/lib/python3.7/site-packages/bcr_api/bwproject.py in request(self, verb, address, params, data)
    182             access_token=self.token,
    183             params=params,
--> 184             data=data,
    185         )
    186 

/opt/conda/lib/python3.7/site-packages/bcr_api/bwproject.py in bare_request(self, verb, address_root, address_suffix, access_token, params, data)
    238                     )
    239                 )
--> 240                 raise RuntimeError(response.json()["errors"])
    241 
    242         logger.debug(response.url)

RuntimeError: [{'code': 0, 'message': "Unrecognized field 'language'"}]