Package Name : azure-servicebus Package Version

Thank you for the feedback <a class="user-mention notranslate" data-hovercard-type="us

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Thanks <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-u

[ServiceBus] Unrecoverable error: "ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.')",about azure/azure-sdk-for-python

Comments (14)

github-actions commented on June 17, 2024

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @EldertGrootenboer.

from azure-sdk-for-python.

kashifkhan commented on June 17, 2024

Thank you for the feedback @saadshaikh3 . Would you be able to provide DEBUG level logs along with setting logging_enable=True - This way we can see the frames as well as the timestamps of activity to further help you out.

import logging
import sys

handler = logging.StreamHandler(stream=sys.stdout)
logger = logging.getLogger('azure.servicebus')
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

...

from azure.servicebus import ServiceBusClient

client = ServiceBusClient(..., logging_enable=True)

from azure-sdk-for-python.

github-actions commented on June 17, 2024

Hi @saadshaikh3. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

from azure-sdk-for-python.

saadshaikh3 commented on June 17, 2024

Hi @kashifkhan. This is the raw stack trace that I have.

AMQPLinkError: null
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py", line 411, in _do_retryable_operation
    return operation(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_servicebus_receiver.py", line 461, in _receive
    receiving = amqp_receive_client.do_work()
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py", line 425, in do_work
    if not self.client_ready():
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py", line 405, in client_ready
    if not self._client_ready():
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py", line 840, in _client_ready
    if self._link.get_state().value != 3:  # ATTACHED
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/link.py", line 116, in get_state
    raise self._error
ServiceBusConnectionError: Link detached unexpectedly. Error condition: amqp:unknown-error.
  File "celery/worker/worker.py", line 202, in start
    self.blueprint.start(self)
  File "celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "celery/worker/consumer/consumer.py", line 340, in start
    blueprint.start(self)
  File "celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "celery/worker/consumer/consumer.py", line 746, in start
    c.loop(*c.loop_args())
  File "celery/worker/loops.py", line 130, in synloop
    connection.drain_events(timeout=2.0)
  File "kombu/connection.py", line 341, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "kombu/transport/virtual/base.py", line 997, in drain_events
    get(self._deliver, timeout=timeout)
  File "kombu/utils/scheduling.py", line 55, in get
    return self.fun(resource, callback, **kwargs)
  File "kombu/transport/virtual/base.py", line 1035, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "kombu/transport/virtual/base.py", line 754, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "kombu/transport/virtual/base.py", line 414, in _poll
    return cycle.get(callback)
  File "kombu/utils/scheduling.py", line 55, in get
    return self.fun(resource, callback, **kwargs)
  File "kombu/transport/virtual/base.py", line 417, in _get_and_deliver
    message = self._get(queue)
  File "kombu/transport/azureservicebus.py", line 266, in _get
    messages = queue_obj.receiver.receive_messages(
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_servicebus_receiver.py", line 702, in receive_messages
    messages: List[ServiceBusReceivedMessage] = self._do_retryable_operation(
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py", line 437, in _do_retryable_operation
    self._backoff(
  File "/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py", line 481, in _backoff
    raise last_exception

This is the grouped exception tree, maybe this could help too :

system

chained-exception

exception

stack-trace

frame

module

azure.servicebus._base_handler

function

_do_retryable_operation

context-line

return operation(**kwargs)

frame

module

azure.servicebus._servicebus_receiver

function

_receive

context-line

receiving = amqp_receive_client.do_work()

frame

module

azure.servicebus._pyamqp.client

function

do_work

context-line

if not self.client_ready():

frame

module

azure.servicebus._pyamqp.client

function

client_ready

context-line

if not self._client_ready():

frame

module

azure.servicebus._pyamqp.client

function

_client_ready

context-line

if self._link.get_state().value != 3: # ATTACHED

frame

module

azure.servicebus._pyamqp.link

function

get_state

context-line

raise self._error

type

AMQPLinkError

exception

stack-trace

frame

module

celery.worker.worker

function

start

context-line

self.blueprint.start(self)

frame

module

celery.bootsteps

function

start

context-line

step.start(parent)

frame

module

celery.bootsteps

function

start

context-line

return self.obj.start()

frame

module

celery.worker.consumer.consumer

function

start

context-line

blueprint.start(self)

frame

module

celery.bootsteps

function

start

context-line

step.start(parent)

frame

module

celery.worker.consumer.consumer

function

start

context-line

c.loop(*c.loop_args())

frame

module

celery.worker.loops

function

synloop

context-line

connection.drain_events(timeout=2.0)

frame

module

kombu.connection

function

drain_events

context-line

return self.transport.drain_events(self.connection, **kwargs)

frame

module

kombu.transport.virtual.base

function

drain_events

context-line

get(self._deliver, timeout=timeout)

frame

module

kombu.utils.scheduling

function

get

context-line

return self.fun(resource, callback, **kwargs)

frame

module

kombu.transport.virtual.base

function

_drain_channel

context-line

return channel.drain_events(callback=callback, timeout=timeout)

frame

module

kombu.transport.virtual.base

function

drain_events

context-line

return self._poll(self.cycle, callback, timeout=timeout)

frame

module

kombu.transport.virtual.base

function

_poll

context-line

return cycle.get(callback)

frame

module

kombu.utils.scheduling

function

get

context-line

return self.fun(resource, callback, **kwargs)

frame

module

kombu.transport.virtual.base

function

_get_and_deliver

context-line

message = self._get(queue)

frame

module

kombu.transport.azureservicebus

function

_get

context-line

messages = queue_obj.receiver.receive_messages(

frame

module

azure.servicebus._servicebus_receiver

function

receive_messages

context-line

messages: List[ServiceBusReceivedMessage] = self._do_retryable_operation(

frame

module

azure.servicebus._base_handler

function

_do_retryable_operation

context-line

self._backoff(

frame

module

azure.servicebus._base_handler

function

_backoff

context-line

raise last_exception

type

ServiceBusConnectionError

from azure-sdk-for-python.

kashifkhan commented on June 17, 2024

@saadshaikh3 I would need the debug logs to understand what the service sent back to the client that it had to detach ( hence frame level logging). The timestamps could also help me understand if there is a timeout issue or something else. Without that itll be difficult to deduce whats happening.

from azure-sdk-for-python.

github-actions commented on June 17, 2024

from azure-sdk-for-python.

github-actions commented on June 17, 2024

Hi @saadshaikh3, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

from azure-sdk-for-python.

saadshaikh3 commented on June 17, 2024

Hi, so the error came again after a week or so. I was able to log it this time with proper timestamps and everything. I have also added Debug level settings in my celery worker so if it comes again, I would be able to attach logs for that as well.

 "5/13/2024, 3:50:19.176 PM",Error," [2024-05-13 15:50:19,175: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:19.176 PM",Error," [2024-05-13 15:50:19,176: INFO/MainProcess] Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:19.176 PM",Error," [2024-05-13 15:50:19,176: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:19.176 PM",Error," [2024-05-13 15:50:19,176: INFO/MainProcess] Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:19.177 PM",Error," [2024-05-13 15:50:19,177: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:19.178 PM",Error," [2024-05-13 15:50:19,178: INFO/MainProcess] Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:19.178 PM",Error," [2024-05-13 15:50:19,178: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:19.178 PM",Error," [2024-05-13 15:50:19,178: INFO/MainProcess] Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:19.462 PM",Error," [2024-05-13 15:50:19,461: INFO/MainProcess] 'servicebus.pysdk-bfc06564' operation has timed out. Last exception before timeout is (ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.'))
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py"", line 411, in _do_retryable_operation
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_servicebus_receiver.py"", line 461, in _receive
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py"", line 425, in do_work
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py"", line 405, in client_ready
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py"", line 840, in _client_ready
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/link.py"", line 116, in get_state
 "5/13/2024, 3:50:19.465 PM",Error," azure.servicebus._pyamqp.error.AMQPLinkError: Error condition: ErrorCondition.UnknownError
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_servicebus_receiver.py"", line 702, in receive_messages
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py"", line 437, in _do_retryable_operation
 "5/13/2024, 3:50:19.465 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py"", line 481, in _backoff
 "5/13/2024, 3:50:19.465 PM",Error," azure.servicebus.exceptions.ServiceBusConnectionError: Link detached unexpectedly. Error condition: amqp:unknown-error.
 "5/13/2024, 3:50:20.440 PM",Error," [2024-05-13 15:50:20,439: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:20.440 PM",Error," [2024-05-13 15:50:20,440: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:20.440 PM",Error," [2024-05-13 15:50:20,440: INFO/MainProcess] Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:20.441 PM",Error," [2024-05-13 15:50:20,440: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:20.441 PM",Error," [2024-05-13 15:50:20,441: INFO/MainProcess] Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
 "5/13/2024, 3:50:20.442 PM",Error," [2024-05-13 15:50:20,442: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:20.442 PM",Error," [2024-05-13 15:50:20,442: INFO/MainProcess] Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:20.442 PM",Error," [2024-05-13 15:50:20,442: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:20.443 PM",Error," [2024-05-13 15:50:20,442: INFO/MainProcess] Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 3:50:20.443 PM",Error," [2024-05-13 15:50:20,443: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
 "5/13/2024, 6:13:59.411 PM",Error," INFO 2024-05-13 18:13:59,410 link 37 134934207375104 Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:13:59.411 PM",Error," INFO 2024-05-13 18:13:59,411 management_link 37 134934207375104 Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:13:59.411 PM",Error," INFO 2024-05-13 18:13:59,411 link 37 134934207375104 Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:13:59.411 PM",Error," INFO 2024-05-13 18:13:59,411 management_link 37 134934207375104 Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:13:59.412 PM",Error," INFO 2024-05-13 18:13:59,412 link 37 134934207375104 Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:13:59.413 PM",Error," INFO 2024-05-13 18:13:59,413 management_link 37 134934207375104 Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:13:59.414 PM",Error," INFO 2024-05-13 18:13:59,413 link 37 134934207375104 Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:13:59.414 PM",Error," INFO 2024-05-13 18:13:59,413 management_link 37 134934207375104 Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:29:51.263 PM",Error," [2024-05-13 18:29:51,262: INFO/MainProcess] Connected to azureservicebus://RootManageSharedAccessKey:**@my-servicebus.servicebus.windows.net
 "5/13/2024, 6:29:51.268 PM",Error," [2024-05-13 18:29:51,268: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
 "5/13/2024, 6:29:51.318 PM",Error," [2024-05-13 18:29:51,318: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
 "5/13/2024, 6:32:47.270 PM",Error," INFO 2024-05-13 18:32:47,270 _universal 37 131869815318272 Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
"5/13/2024, 6:32:47.304 PM",Error," INFO 2024-05-13 18:32:47,304 _universal 37 131869815318272 Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
"5/13/2024, 6:32:47.440 PM",Error," [2024-05-13 18:32:47,440: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
"5/13/2024, 6:24:47.751 PM",Error," [2024-05-13 18:24:47,750: INFO/MainProcess] Connected to azureservicebus://RootManageSharedAccessKey:**@my-servicebus.servicebus.windows.net
"5/13/2024, 6:24:47.757 PM",Error," [2024-05-13 18:24:47,757: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
"5/13/2024, 6:24:47.798 PM",Error," [2024-05-13 18:24:47,798: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
"5/13/2024, 6:25:24.930 PM",Error," [2024-05-13 18:25:24,930: INFO/MainProcess] Connected to azureservicebus://RootManageSharedAccessKey:**@my-servicebus.servicebus.windows.net
 "5/13/2024, 6:25:24.935 PM",Error," [2024-05-13 18:25:24,935: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
 "5/13/2024, 6:25:24.970 PM",Error," [2024-05-13 18:25:24,970: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
 "5/13/2024, 6:25:48.740 PM",Error," [2024-05-13 18:25:48,739: INFO/MainProcess] Connected to azureservicebus://RootManageSharedAccessKey:**@my-servicebus.servicebus.windows.net
 "5/13/2024, 6:25:48.746 PM",Error," [2024-05-13 18:25:48,745: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
 "5/13/2024, 6:25:48.781 PM",Error," [2024-05-13 18:25:48,780: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
 "5/13/2024, 6:25:40.129 PM",Error," [2024-05-13 18:25:40,129: INFO/MainProcess] Connected to azureservicebus://RootManageSharedAccessKey:**@my-servicebus.servicebus.windows.net
"5/13/2024, 6:25:40.136 PM",Error," [2024-05-13 18:25:40,135: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
"5/13/2024, 6:25:40.185 PM",Error," [2024-05-13 18:25:40,185: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
"5/13/2024, 6:05:54.637 PM",Error," INFO 2024-05-13 18:05:54,637 _universal 37 134934207375104 Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
"5/13/2024, 6:05:54.670 PM",Error," INFO 2024-05-13 18:05:54,670 _universal 37 134934207375104 Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
"5/13/2024, 6:05:54.786 PM",Error," [2024-05-13 18:05:54,785: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
"5/13/2024, 5:57:55.097 PM",Error," [2024-05-13 17:57:55,096: INFO/MainProcess] Connected to azureservicebus://RootManageSharedAccessKey:**@my-servicebus.servicebus.windows.net
"5/13/2024, 5:57:55.103 PM",Error," [2024-05-13 17:57:55,103: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?api-version=REDACTED'
"5/13/2024, 5:57:55.136 PM",Error," [2024-05-13 17:57:55,136: INFO/MainProcess] Request URL: 'https://my-servicebus.servicebus.windows.net/celery-queue?enrich=REDACTED&api-version=REDACTED'
"5/13/2024, 6:12:21.001 PM",Error," [2024-05-13 18:12:21,001: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:21.001 PM",Error," [2024-05-13 18:12:21,001: INFO/MainProcess] Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:21.002 PM",Error," [2024-05-13 18:12:21,001: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:21.002 PM",Error," [2024-05-13 18:12:21,002: INFO/MainProcess] Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:21.003 PM",Error," [2024-05-13 18:12:21,003: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:21.003 PM",Error," [2024-05-13 18:12:21,003: INFO/MainProcess] Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:21.003 PM",Error," [2024-05-13 18:12:21,003: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:21.004 PM",Error," [2024-05-13 18:12:21,004: INFO/MainProcess] Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:22.622 PM",Error," [2024-05-13 18:12:22,622: INFO/MainProcess] 'servicebus.pysdk-113b18c7' has an exception (ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.')). Retrying...
"5/13/2024, 6:12:43.546 PM",Error," [2024-05-13 18:12:43,545: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:43.546 PM",Error," [2024-05-13 18:12:43,546: INFO/MainProcess] Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:43.546 PM",Error," [2024-05-13 18:12:43,546: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:43.547 PM",Error," [2024-05-13 18:12:43,546: INFO/MainProcess] Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:43.548 PM",Error," [2024-05-13 18:12:43,548: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:43.548 PM",Error," [2024-05-13 18:12:43,548: INFO/MainProcess] Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:43.548 PM",Error," [2024-05-13 18:12:43,548: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:43.548 PM",Error," [2024-05-13 18:12:43,548: INFO/MainProcess] Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:43.739 PM",Error," [2024-05-13 18:12:43,739: INFO/MainProcess] 'servicebus.pysdk-113b18c7' operation has timed out. Last exception before timeout is (ServiceBusError('The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:d616a687f7394b929c90331d0e71810c_G7, SystemTracker:gateway7, Timestamp:2024-05-13T18:12:43 Error condition: amqp:internal-error.'))
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py"", line 411, in _do_retryable_operation
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_servicebus_receiver.py"", line 416, in _receive
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_servicebus_receiver.py"", line 400, in _open
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py"", line 405, in client_ready
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/client.py"", line 840, in _client_ready
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_pyamqp/link.py"", line 116, in get_state
"5/13/2024, 6:12:43.743 PM",Error," azure.servicebus._pyamqp.error.AMQPLinkError: Error condition: amqp:internal-error
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_servicebus_receiver.py"", line 702, in receive_messages
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py"", line 437, in _do_retryable_operation
"5/13/2024, 6:12:43.743 PM",Error,"   File ""/usr/local/lib/python3.10/site-packages/azure/servicebus/_base_handler.py"", line 481, in _backoff
"5/13/2024, 6:12:43.743 PM",Error," azure.servicebus.exceptions.ServiceBusError: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:d616a687f7394b929c90331d0e71810c_G7, SystemTracker:gateway7, Timestamp:2024-05-13T18:12:43 Error condition: amqp:internal-error.
"5/13/2024, 6:12:44.283 PM",Error," [2024-05-13 18:12:44,283: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:44.284 PM",Error," [2024-05-13 18:12:44,284: INFO/MainProcess] Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:44.284 PM",Error," [2024-05-13 18:12:44,284: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:44.284 PM",Error," [2024-05-13 18:12:44,284: INFO/MainProcess] Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
"5/13/2024, 6:12:44.286 PM",Error," [2024-05-13 18:12:44,286: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:44.286 PM",Error," [2024-05-13 18:12:44,286: INFO/MainProcess] Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:44.286 PM",Error," [2024-05-13 18:12:44,286: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
"5/13/2024, 6:12:44.286 PM",Error," [2024-05-13 18:12:44,286: INFO/MainProcess] Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>"5/13/2024, 6:14:01.508 PM",Error," INFO 2024-05-13 18:14:01,508 _base_handler 37 134934207375104 'servicebus.pysdk-36d47169' has an exception (ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.')). Retrying...

from azure-sdk-for-python.

saadshaikh3 commented on June 17, 2024

Hi, the error came again and here are the debug level logs along with timestamps. I hope this will help in resolving the issue

"TimeGenerated [UTC]",ResultDescription
5/29/2024, 12:50:25.025 PM"," [2024-05-29 12:50:25,025: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:25.227 PM"," [2024-05-29 12:50:25,227: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:25.428 PM"," [2024-05-29 12:50:25,428: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:25.461 PM"," [2024-05-29 12:50:25,461: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9956310549750924 secs.
5/29/2024, 12:50:25.630 PM"," [2024-05-29 12:50:25,630: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:25.832 PM"," [2024-05-29 12:50:25,831: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:26.033 PM"," [2024-05-29 12:50:26,033: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:26.234 PM"," [2024-05-29 12:50:26,234: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:26.436 PM"," [2024-05-29 12:50:26,436: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:26.459 PM"," [2024-05-29 12:50:26,459: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:26.637 PM"," [2024-05-29 12:50:26,637: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:26.839 PM"," [2024-05-29 12:50:26,839: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:27.040 PM"," [2024-05-29 12:50:27,040: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:27.241 PM"," [2024-05-29 12:50:27,241: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:27.465 PM"," [2024-05-29 12:50:27,464: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9947724230587482 secs.
5/29/2024, 12:50:27.644 PM"," [2024-05-29 12:50:27,644: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:27.846 PM"," [2024-05-29 12:50:27,845: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:28.047 PM"," [2024-05-29 12:50:28,047: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:28.249 PM"," [2024-05-29 12:50:28,248: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:28.450 PM"," [2024-05-29 12:50:28,450: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:28.460 PM"," [2024-05-29 12:50:28,460: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:28.652 PM"," [2024-05-29 12:50:28,651: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:28.853 PM"," [2024-05-29 12:50:28,853: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:29.054 PM"," [2024-05-29 12:50:29,054: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:29.256 PM"," [2024-05-29 12:50:29,256: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:29.458 PM"," [2024-05-29 12:50:29,457: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:29.461 PM"," [2024-05-29 12:50:29,460: DEBUG/MainProcess] Timer wake-up! Next ETA 0.999704672023654 secs.
5/29/2024, 12:50:29.659 PM"," [2024-05-29 12:50:29,658: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:29.860 PM"," [2024-05-29 12:50:29,860: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:30.062 PM"," [2024-05-29 12:50:30,061: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:30.263 PM"," [2024-05-29 12:50:30,263: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:30.465 PM"," [2024-05-29 12:50:30,464: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:30.466 PM"," [2024-05-29 12:50:30,465: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:30.666 PM"," [2024-05-29 12:50:30,665: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:30.867 PM"," [2024-05-29 12:50:30,867: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:31.069 PM"," [2024-05-29 12:50:31,068: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:31.270 PM"," [2024-05-29 12:50:31,270: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:31.470 PM"," [2024-05-29 12:50:31,470: DEBUG/MainProcess] Timer wake-up! Next ETA 0.995464236009866 secs.
5/29/2024, 12:50:31.471 PM"," [2024-05-29 12:50:31,471: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:31.673 PM"," [2024-05-29 12:50:31,673: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:31.874 PM"," [2024-05-29 12:50:31,874: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:32.076 PM"," [2024-05-29 12:50:32,076: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:32.277 PM"," [2024-05-29 12:50:32,277: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:32.470 PM"," [2024-05-29 12:50:32,469: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:32.479 PM"," [2024-05-29 12:50:32,479: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:32.882 PM"," [2024-05-29 12:50:32,882: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:33.083 PM"," [2024-05-29 12:50:33,083: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:33.285 PM"," [2024-05-29 12:50:33,285: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:33.471 PM"," [2024-05-29 12:50:33,471: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9986437428742647 secs.
5/29/2024, 12:50:33.486 PM"," [2024-05-29 12:50:33,486: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:33.688 PM"," [2024-05-29 12:50:33,688: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:33.890 PM"," [2024-05-29 12:50:33,889: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:34.091 PM"," [2024-05-29 12:50:34,091: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:34.293 PM"," [2024-05-29 12:50:34,292: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:34.476 PM"," [2024-05-29 12:50:34,475: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:34.494 PM"," [2024-05-29 12:50:34,494: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:34.696 PM"," [2024-05-29 12:50:34,695: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:34.897 PM"," [2024-05-29 12:50:34,897: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:35.099 PM"," [2024-05-29 12:50:35,099: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:35.300 PM"," [2024-05-29 12:50:35,300: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:35.478 PM"," [2024-05-29 12:50:35,478: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9976736069656909 secs.
5/29/2024, 12:50:35.502 PM"," [2024-05-29 12:50:35,501: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:35.703 PM"," [2024-05-29 12:50:35,703: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:35.905 PM"," [2024-05-29 12:50:35,904: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:36.106 PM"," [2024-05-29 12:50:36,106: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:36.308 PM"," [2024-05-29 12:50:36,307: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:36.404 PM"," [2024-05-29 12:50:36,404: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:36.481 PM"," [2024-05-29 12:50:36,481: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:36.606 PM"," [2024-05-29 12:50:36,606: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:36.808 PM"," [2024-05-29 12:50:36,807: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:37.009 PM"," [2024-05-29 12:50:37,009: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:37.211 PM"," [2024-05-29 12:50:37,210: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:37.412 PM"," [2024-05-29 12:50:37,412: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:37.487 PM"," [2024-05-29 12:50:37,486: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9947146091144532 secs.
5/29/2024, 12:50:37.614 PM"," [2024-05-29 12:50:37,613: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:37.815 PM"," [2024-05-29 12:50:37,815: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:38.218 PM"," [2024-05-29 12:50:38,218: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:38.420 PM"," [2024-05-29 12:50:38,420: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:38.483 PM"," [2024-05-29 12:50:38,482: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:38.622 PM"," [2024-05-29 12:50:38,621: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:38.823 PM"," [2024-05-29 12:50:38,823: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:39.025 PM"," [2024-05-29 12:50:39,024: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:39.226 PM"," [2024-05-29 12:50:39,226: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:39.428 PM"," [2024-05-29 12:50:39,427: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:39.488 PM"," [2024-05-29 12:50:39,488: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9946552978362888 secs.
5/29/2024, 12:50:39.629 PM"," [2024-05-29 12:50:39,629: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:39.831 PM"," [2024-05-29 12:50:39,830: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:40.032 PM"," [2024-05-29 12:50:40,032: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:40.234 PM"," [2024-05-29 12:50:40,233: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:40.435 PM"," [2024-05-29 12:50:40,435: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:40.488 PM"," [2024-05-29 12:50:40,488: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:40.637 PM"," [2024-05-29 12:50:40,637: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:40.838 PM"," [2024-05-29 12:50:40,838: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:41.040 PM"," [2024-05-29 12:50:41,040: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:41.242 PM"," [2024-05-29 12:50:41,241: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:41.443 PM"," [2024-05-29 12:50:41,443: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:41.489 PM"," [2024-05-29 12:50:41,489: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9984966781921685 secs.
5/29/2024, 12:50:41.644 PM"," [2024-05-29 12:50:41,644: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:41.846 PM"," [2024-05-29 12:50:41,845: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:42.047 PM"," [2024-05-29 12:50:42,047: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:42.248 PM"," [2024-05-29 12:50:42,248: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:42.450 PM"," [2024-05-29 12:50:42,449: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:42.493 PM"," [2024-05-29 12:50:42,493: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:42.651 PM"," [2024-05-29 12:50:42,651: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:42.853 PM"," [2024-05-29 12:50:42,852: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:43.054 PM"," [2024-05-29 12:50:43,054: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:43.458 PM"," [2024-05-29 12:50:43,457: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:43.495 PM"," [2024-05-29 12:50:43,495: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9985794760286808 secs.
5/29/2024, 12:50:43.659 PM"," [2024-05-29 12:50:43,659: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:43.860 PM"," [2024-05-29 12:50:43,860: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:44.062 PM"," [2024-05-29 12:50:44,062: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:44.263 PM"," [2024-05-29 12:50:44,263: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:44.465 PM"," [2024-05-29 12:50:44,465: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:44.498 PM"," [2024-05-29 12:50:44,498: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:44.666 PM"," [2024-05-29 12:50:44,666: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:44.868 PM"," [2024-05-29 12:50:44,868: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:45.069 PM"," [2024-05-29 12:50:45,069: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:45.271 PM"," [2024-05-29 12:50:45,271: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:45.472 PM"," [2024-05-29 12:50:45,472: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:45.503 PM"," [2024-05-29 12:50:45,503: DEBUG/MainProcess] Timer wake-up! Next ETA 0.99477185215801 secs.
5/29/2024, 12:50:45.673 PM"," [2024-05-29 12:50:45,673: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:45.875 PM"," [2024-05-29 12:50:45,875: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:46.077 PM"," [2024-05-29 12:50:46,076: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:46.278 PM"," [2024-05-29 12:50:46,278: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:46.479 PM"," [2024-05-29 12:50:46,479: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:46.503 PM"," [2024-05-29 12:50:46,502: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
5/29/2024, 12:50:46.681 PM"," [2024-05-29 12:50:46,681: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:46.883 PM"," [2024-05-29 12:50:46,882: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:47.084 PM"," [2024-05-29 12:50:47,084: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:47.285 PM"," [2024-05-29 12:50:47,285: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:47.487 PM"," [2024-05-29 12:50:47,487: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:47.505 PM"," [2024-05-29 12:50:47,505: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9973496710881591 secs.
5/29/2024, 12:50:47.688 PM"," [2024-05-29 12:50:47,688: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:47.890 PM"," [2024-05-29 12:50:47,889: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:47.935 PM"," [2024-05-29 12:50:47,935: DEBUG/MainProcess] CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
5/29/2024, 12:50:47.935 PM"," [2024-05-29 12:50:47,935: INFO/MainProcess] AMQP error occurred: (AMQPLinkError('Error condition: ErrorCondition.UnknownError\n Error Description: Link detached unexpectedly.')), condition: (<ErrorCondition.UnknownError: b'amqp:unknown-error'>), description: ('Link detached unexpectedly.').
5/29/2024, 12:50:47.935 PM"," [2024-05-29 12:50:47,935: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:50:47.935 PM"," [2024-05-29 12:50:47,935: INFO/MainProcess] Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:50:47.936 PM"," [2024-05-29 12:50:47,935: INFO/MainProcess] Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:50:47.936 PM"," [2024-05-29 12:50:47,936: INFO/MainProcess] Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:50:47.937 PM"," [2024-05-29 12:50:47,937: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:50:47.937 PM"," [2024-05-29 12:50:47,937: INFO/MainProcess] Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:50:47.937 PM"," [2024-05-29 12:50:47,937: INFO/MainProcess] Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:50:47.937 PM"," [2024-05-29 12:50:47,937: INFO/MainProcess] Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:50:48.276 PM"," [2024-05-29 12:50:48,275: INFO/MainProcess] 'servicebus.pysdk-d7ce1c7d' operation has timed out. Last exception before timeout is (ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.'))
5/29/2024, 12:50:48.279 PM"," [2024-05-29 12:50:48,276: CRITICAL/MainProcess] Unrecoverable error: ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.')
5/29/2024, 12:50:48.279 PM","     receiving = amqp_receive_client.do_work()
5/29/2024, 12:50:48.280 PM"," azure.servicebus.exceptions.ServiceBusConnectionError: Link detached unexpectedly. Error condition: amqp:unknown-error.
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,477: DEBUG/MainProcess] | Worker: Closing Pool...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,477: DEBUG/MainProcess] | Worker: Closing Consumer...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Worker: Stopping Consumer...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Closing Connection...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Closing Events...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Closing Heart...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Closing Tasks...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Closing event loop...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Stopping event loop...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Stopping Tasks...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] Canceling task consumer...
5/29/2024, 12:50:48.478 PM"," [2024-05-29 12:50:48,478: DEBUG/MainProcess] | Consumer: Stopping Heart...
5/29/2024, 12:50:48.479 PM"," [2024-05-29 12:50:48,479: DEBUG/MainProcess] | Consumer: Stopping Events...
5/29/2024, 12:50:48.479 PM"," [2024-05-29 12:50:48,479: DEBUG/MainProcess] | Consumer: Stopping Connection...
5/29/2024, 12:50:48.479 PM"," [2024-05-29 12:50:48,479: DEBUG/MainProcess] | Worker: Stopping Pool...
5/29/2024, 12:50:48.508 PM"," [2024-05-29 12:50:48,508: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
5/29/2024, 12:50:48.508 PM"," [2024-05-29 12:50:48,508: DEBUG/MainProcess] Canceling task consumer...
5/29/2024, 12:50:48.508 PM"," [2024-05-29 12:50:48,508: DEBUG/MainProcess] Closing consumer channel...
5/29/2024, 12:50:48.508 PM"," [2024-05-29 12:50:48,508: DEBUG/MainProcess] | Consumer: Shutdown Heart...
5/29/2024, 12:50:48.509 PM"," [2024-05-29 12:50:48,508: DEBUG/MainProcess] | Consumer: Shutdown Events...
5/29/2024, 12:50:48.509 PM"," [2024-05-29 12:50:48,509: DEBUG/MainProcess] | Consumer: Shutdown Connection...
5/29/2024, 12:50:48.516 PM"," [2024-05-29 12:50:48,516: DEBUG/MainProcess] https://o203083.ingest.sentry.io:443 ""POST /api/4505636418158592/envelope/ HTTP/1.1"" 200 0
5/29/2024, 12:51:07.969 PM"," DEBUG 2024-05-29 12:51:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 12:51:07.969 PM"," DEBUG 2024-05-29 12:51:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 05:53:07.968569-07:00 (in 119.999093 seconds)
5/29/2024, 12:53:07.969 PM"," DEBUG 2024-05-29 12:53:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 12:53:07.969 PM"," DEBUG 2024-05-29 12:53:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 05:55:07.968569-07:00 (in 119.999031 seconds)
5/29/2024, 12:55:07.969 PM"," DEBUG 2024-05-29 12:55:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 12:55:07.969 PM"," DEBUG 2024-05-29 12:55:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 05:55:07.971804-07:00 (in 0.002308 seconds)
5/29/2024, 12:55:07.972 PM"," DEBUG 2024-05-29 12:55:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 12:55:07.972 PM"," DEBUG 2024-05-29 12:55:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 05:57:07.968569-07:00 (in 119.996175 seconds)
5/29/2024, 12:55:13.259 PM"," INFO 2024-05-29 12:55:13,259 _pyamqp_transport 52 123216539801344 AMQP error occurred: (AMQPLinkError('Error condition: ErrorCondition.UnknownError\n Error Description: Link detached unexpectedly.')), condition: (<ErrorCondition.UnknownError: b'amqp:unknown-error'>), description: ('Link detached unexpectedly.').
5/29/2024, 12:55:13.259 PM"," INFO 2024-05-29 12:55:13,259 link 52 123216539801344 Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:55:13.260 PM"," INFO 2024-05-29 12:55:13,259 management_link 52 123216539801344 Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:55:13.260 PM"," INFO 2024-05-29 12:55:13,260 link 52 123216539801344 Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:55:13.260 PM"," INFO 2024-05-29 12:55:13,260 management_link 52 123216539801344 Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 12:55:13.261 PM"," INFO 2024-05-29 12:55:13,261 link 52 123216539801344 Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:55:13.262 PM"," INFO 2024-05-29 12:55:13,262 management_link 52 123216539801344 Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:55:13.262 PM"," INFO 2024-05-29 12:55:13,262 link 52 123216539801344 Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:55:13.262 PM"," INFO 2024-05-29 12:55:13,262 management_link 52 123216539801344 Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 12:55:14.873 PM"," INFO 2024-05-29 12:55:14,873 _base_handler 52 123216539801344 'servicebus.pysdk-76843de2' has an exception (ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.')). Retrying...
5/29/2024, 12:57:07.971 PM"," DEBUG 2024-05-29 12:57:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 12:57:07.971 PM"," DEBUG 2024-05-29 12:57:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 05:59:07.968569-07:00 (in 119.998712 seconds)
5/29/2024, 12:59:07.969 PM"," DEBUG 2024-05-29 12:59:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 12:59:07.969 PM"," DEBUG 2024-05-29 12:59:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:01:07.968569-07:00 (in 119.999075 seconds)
5/29/2024, 1:01:07.969 PM"," DEBUG 2024-05-29 13:01:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:01:07.969 PM"," DEBUG 2024-05-29 13:01:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:03:07.968569-07:00 (in 119.999047 seconds)
5/29/2024, 1:03:07.969 PM"," DEBUG 2024-05-29 13:03:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:03:07.969 PM"," DEBUG 2024-05-29 13:03:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:03:07.971804-07:00 (in 0.002452 seconds)
5/29/2024, 1:03:07.972 PM"," DEBUG 2024-05-29 13:03:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:03:07.973 PM"," DEBUG 2024-05-29 13:03:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:05:07.968569-07:00 (in 119.996078 seconds)
5/29/2024, 1:05:07.969 PM"," DEBUG 2024-05-29 13:05:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:05:07.969 PM"," DEBUG 2024-05-29 13:05:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:07:07.968569-07:00 (in 119.999084 seconds)
5/29/2024, 1:07:07.969 PM"," DEBUG 2024-05-29 13:07:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:07:07.969 PM"," DEBUG 2024-05-29 13:07:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:09:07.968569-07:00 (in 119.999274 seconds)
5/29/2024, 1:09:07.969 PM"," DEBUG 2024-05-29 13:09:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:09:07.969 PM"," DEBUG 2024-05-29 13:09:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:11:07.968569-07:00 (in 119.999170 seconds)
5/29/2024, 1:11:07.969 PM"," DEBUG 2024-05-29 13:11:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:11:07.969 PM"," DEBUG 2024-05-29 13:11:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:11:07.971804-07:00 (in 0.002298 seconds)
5/29/2024, 1:11:07.972 PM"," DEBUG 2024-05-29 13:11:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:11:07.972 PM"," DEBUG 2024-05-29 13:11:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:13:07.968569-07:00 (in 119.996122 seconds)
5/29/2024, 1:13:07.969 PM"," DEBUG 2024-05-29 13:13:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:13:07.969 PM"," DEBUG 2024-05-29 13:13:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:15:07.968569-07:00 (in 119.999108 seconds)
5/29/2024, 1:15:07.969 PM"," DEBUG 2024-05-29 13:15:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:15:07.969 PM"," DEBUG 2024-05-29 13:15:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:17:07.968569-07:00 (in 119.999082 seconds)
5/29/2024, 1:17:07.977 PM"," DEBUG 2024-05-29 13:17:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:17:07.977 PM"," DEBUG 2024-05-29 13:17:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:19:07.968569-07:00 (in 119.999269 seconds)
5/29/2024, 1:19:07.969 PM"," DEBUG 2024-05-29 13:19:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:19:07.970 PM"," DEBUG 2024-05-29 13:19:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:19:07.971804-07:00 (in 0.002141 seconds)
5/29/2024, 1:19:07.972 PM"," DEBUG 2024-05-29 13:19:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:19:07.972 PM"," DEBUG 2024-05-29 13:19:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:21:07.968569-07:00 (in 119.996063 seconds)
5/29/2024, 1:21:07.969 PM"," DEBUG 2024-05-29 13:21:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:21:07.969 PM"," DEBUG 2024-05-29 13:21:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:23:07.968569-07:00 (in 119.999106 seconds)
5/29/2024, 1:23:07.969 PM"," DEBUG 2024-05-29 13:23:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:23:07.969 PM"," DEBUG 2024-05-29 13:23:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:25:07.968569-07:00 (in 119.999067 seconds)
5/29/2024, 1:25:07.969 PM"," DEBUG 2024-05-29 13:25:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:25:07.969 PM"," DEBUG 2024-05-29 13:25:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:27:07.968569-07:00 (in 119.999000 seconds)
5/29/2024, 1:27:07.969 PM"," DEBUG 2024-05-29 13:27:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:27:07.969 PM"," DEBUG 2024-05-29 13:27:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:27:07.971804-07:00 (in 0.002368 seconds)
5/29/2024, 1:27:07.972 PM"," DEBUG 2024-05-29 13:27:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:27:07.972 PM"," DEBUG 2024-05-29 13:27:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:29:07.968569-07:00 (in 119.996098 seconds)
5/29/2024, 1:29:07.969 PM"," DEBUG 2024-05-29 13:29:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:29:07.969 PM"," DEBUG 2024-05-29 13:29:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:31:07.968569-07:00 (in 119.999053 seconds)
5/29/2024, 1:31:07.969 PM"," DEBUG 2024-05-29 13:31:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:31:07.969 PM"," DEBUG 2024-05-29 13:31:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:33:07.968569-07:00 (in 119.999250 seconds)
5/29/2024, 1:33:07.969 PM"," DEBUG 2024-05-29 13:33:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:33:07.969 PM"," DEBUG 2024-05-29 13:33:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:35:07.968569-07:00 (in 119.999085 seconds)
5/29/2024, 1:35:07.969 PM"," DEBUG 2024-05-29 13:35:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:35:07.969 PM"," DEBUG 2024-05-29 13:35:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:35:07.971804-07:00 (in 0.002378 seconds)
5/29/2024, 1:35:07.972 PM"," DEBUG 2024-05-29 13:35:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:35:07.973 PM"," DEBUG 2024-05-29 13:35:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:37:07.968569-07:00 (in 119.995819 seconds)
5/29/2024, 1:37:07.969 PM"," DEBUG 2024-05-29 13:37:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:37:07.969 PM"," DEBUG 2024-05-29 13:37:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:39:07.968569-07:00 (in 119.999017 seconds)
5/29/2024, 1:39:07.969 PM"," DEBUG 2024-05-29 13:39:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:39:07.969 PM"," DEBUG 2024-05-29 13:39:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:41:07.968569-07:00 (in 119.999149 seconds)
5/29/2024, 1:41:07.969 PM"," DEBUG 2024-05-29 13:41:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:41:07.969 PM"," DEBUG 2024-05-29 13:41:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:43:07.968569-07:00 (in 119.999015 seconds)
5/29/2024, 1:43:07.969 PM"," DEBUG 2024-05-29 13:43:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:43:07.969 PM"," DEBUG 2024-05-29 13:43:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:43:07.971804-07:00 (in 0.002370 seconds)
5/29/2024, 1:43:07.972 PM"," DEBUG 2024-05-29 13:43:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:43:07.973 PM"," DEBUG 2024-05-29 13:43:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:45:07.968569-07:00 (in 119.996096 seconds)
5/29/2024, 1:45:07.969 PM"," DEBUG 2024-05-29 13:45:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:45:07.969 PM"," DEBUG 2024-05-29 13:45:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:47:07.968569-07:00 (in 119.999054 seconds)
5/29/2024, 1:47:07.969 PM"," DEBUG 2024-05-29 13:47:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:47:07.969 PM"," DEBUG 2024-05-29 13:47:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:49:07.968569-07:00 (in 119.999223 seconds)
5/29/2024, 1:49:07.969 PM"," DEBUG 2024-05-29 13:49:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:49:07.969 PM"," DEBUG 2024-05-29 13:49:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:51:07.968569-07:00 (in 119.999188 seconds)
5/29/2024, 1:51:07.969 PM"," DEBUG 2024-05-29 13:51:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:51:07.970 PM"," DEBUG 2024-05-29 13:51:07,970 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:51:07.971804-07:00 (in 0.001771 seconds)
5/29/2024, 1:51:07.972 PM"," DEBUG 2024-05-29 13:51:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:51:07.973 PM"," DEBUG 2024-05-29 13:51:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:53:07.968569-07:00 (in 119.996166 seconds)
5/29/2024, 1:53:07.969 PM"," DEBUG 2024-05-29 13:53:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:53:07.969 PM"," DEBUG 2024-05-29 13:53:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:55:07.968569-07:00 (in 119.999184 seconds)
5/29/2024, 1:55:07.969 PM"," DEBUG 2024-05-29 13:55:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:55:07.969 PM"," DEBUG 2024-05-29 13:55:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:57:07.968569-07:00 (in 119.999070 seconds)
5/29/2024, 1:57:07.969 PM"," DEBUG 2024-05-29 13:57:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:57:07.969 PM"," DEBUG 2024-05-29 13:57:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:59:07.968569-07:00 (in 119.999220 seconds)
5/29/2024, 1:59:07.969 PM"," DEBUG 2024-05-29 13:59:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:59:07.969 PM"," DEBUG 2024-05-29 13:59:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 06:59:07.971804-07:00 (in 0.002376 seconds)
5/29/2024, 1:59:07.972 PM"," DEBUG 2024-05-29 13:59:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 1:59:07.972 PM"," DEBUG 2024-05-29 13:59:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:01:07.968569-07:00 (in 119.996108 seconds)
5/29/2024, 2:01:07.969 PM"," DEBUG 2024-05-29 14:01:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:01:07.969 PM"," DEBUG 2024-05-29 14:01:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:03:07.968569-07:00 (in 119.999141 seconds)
5/29/2024, 2:03:07.969 PM"," DEBUG 2024-05-29 14:03:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:03:07.969 PM"," DEBUG 2024-05-29 14:03:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:05:07.968569-07:00 (in 119.999031 seconds)
5/29/2024, 2:05:07.969 PM"," DEBUG 2024-05-29 14:05:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:05:07.969 PM"," DEBUG 2024-05-29 14:05:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:07:07.968569-07:00 (in 119.999056 seconds)
5/29/2024, 2:07:07.969 PM"," DEBUG 2024-05-29 14:07:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:07:07.969 PM"," DEBUG 2024-05-29 14:07:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:07:07.971804-07:00 (in 0.002311 seconds)
5/29/2024, 2:07:07.972 PM"," DEBUG 2024-05-29 14:07:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:07:07.972 PM"," DEBUG 2024-05-29 14:07:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:09:07.968569-07:00 (in 119.995875 seconds)
5/29/2024, 2:09:07.969 PM"," DEBUG 2024-05-29 14:09:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:09:07.969 PM"," DEBUG 2024-05-29 14:09:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:11:07.968569-07:00 (in 119.998961 seconds)
5/29/2024, 2:11:07.969 PM"," DEBUG 2024-05-29 14:11:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:11:07.969 PM"," DEBUG 2024-05-29 14:11:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:13:07.968569-07:00 (in 119.999195 seconds)
5/29/2024, 2:13:07.969 PM"," DEBUG 2024-05-29 14:13:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:13:07.969 PM"," DEBUG 2024-05-29 14:13:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:15:07.968569-07:00 (in 119.999010 seconds)
5/29/2024, 2:15:07.969 PM"," DEBUG 2024-05-29 14:15:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:15:07.969 PM"," DEBUG 2024-05-29 14:15:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:15:07.971804-07:00 (in 0.002261 seconds)
5/29/2024, 2:15:07.972 PM"," DEBUG 2024-05-29 14:15:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:15:07.972 PM"," DEBUG 2024-05-29 14:15:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:17:07.968569-07:00 (in 119.996101 seconds)
5/29/2024, 2:17:07.969 PM"," DEBUG 2024-05-29 14:17:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:17:07.969 PM"," DEBUG 2024-05-29 14:17:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:19:07.968569-07:00 (in 119.998954 seconds)
5/29/2024, 2:19:07.969 PM"," DEBUG 2024-05-29 14:19:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:19:07.969 PM"," DEBUG 2024-05-29 14:19:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:21:07.968569-07:00 (in 119.999155 seconds)
5/29/2024, 2:21:07.969 PM"," DEBUG 2024-05-29 14:21:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:21:07.969 PM"," DEBUG 2024-05-29 14:21:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:23:07.968569-07:00 (in 119.999055 seconds)
5/29/2024, 2:23:07.969 PM"," DEBUG 2024-05-29 14:23:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:23:07.969 PM"," DEBUG 2024-05-29 14:23:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:23:07.971804-07:00 (in 0.002449 seconds)
5/29/2024, 2:23:07.972 PM"," DEBUG 2024-05-29 14:23:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:23:07.973 PM"," DEBUG 2024-05-29 14:23:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:25:07.968569-07:00 (in 119.996129 seconds)
5/29/2024, 2:23:12.789 PM"," INFO 2024-05-29 14:23:12,789 _pyamqp_transport 52 123216539801344 AMQP error occurred: (AMQPLinkError('Error condition: ErrorCondition.UnknownError\n Error Description: Link detached unexpectedly.')), condition: (<ErrorCondition.UnknownError: b'amqp:unknown-error'>), description: ('Link detached unexpectedly.').
5/29/2024, 2:23:12.790 PM"," INFO 2024-05-29 14:23:12,789 link 52 123216539801344 Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 2:23:12.790 PM"," INFO 2024-05-29 14:23:12,790 management_link 52 123216539801344 Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 2:23:12.790 PM"," INFO 2024-05-29 14:23:12,790 link 52 123216539801344 Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 2:23:12.790 PM"," INFO 2024-05-29 14:23:12,790 management_link 52 123216539801344 Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
5/29/2024, 2:23:12.792 PM"," INFO 2024-05-29 14:23:12,792 link 52 123216539801344 Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 2:23:12.792 PM"," INFO 2024-05-29 14:23:12,792 management_link 52 123216539801344 Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 2:23:12.792 PM"," INFO 2024-05-29 14:23:12,792 link 52 123216539801344 Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 2:23:12.792 PM"," INFO 2024-05-29 14:23:12,792 management_link 52 123216539801344 Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
5/29/2024, 2:23:14.865 PM"," INFO 2024-05-29 14:23:14,864 _base_handler 52 123216539801344 'servicebus.pysdk-76843de2' has an exception (ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.')). Retrying...
5/29/2024, 2:25:07.969 PM"," DEBUG 2024-05-29 14:25:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:25:07.969 PM"," DEBUG 2024-05-29 14:25:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:27:07.968569-07:00 (in 119.999093 seconds)
5/29/2024, 2:27:07.969 PM"," DEBUG 2024-05-29 14:27:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:27:07.969 PM"," DEBUG 2024-05-29 14:27:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:29:07.968569-07:00 (in 119.999056 seconds)
5/29/2024, 2:29:07.969 PM"," DEBUG 2024-05-29 14:29:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:29:07.969 PM"," DEBUG 2024-05-29 14:29:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:31:07.968569-07:00 (in 119.998971 seconds)
5/29/2024, 2:31:07.969 PM"," DEBUG 2024-05-29 14:31:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:31:07.969 PM"," DEBUG 2024-05-29 14:31:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:31:07.971804-07:00 (in 0.002199 seconds)
5/29/2024, 2:31:07.972 PM"," DEBUG 2024-05-29 14:31:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:31:07.972 PM"," DEBUG 2024-05-29 14:31:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:33:07.968569-07:00 (in 119.996037 seconds)
5/29/2024, 2:33:07.969 PM"," DEBUG 2024-05-29 14:33:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:33:07.969 PM"," DEBUG 2024-05-29 14:33:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:35:07.968569-07:00 (in 119.998910 seconds)
5/29/2024, 2:35:07.969 PM"," DEBUG 2024-05-29 14:35:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:35:07.969 PM"," DEBUG 2024-05-29 14:35:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:37:07.968569-07:00 (in 119.999008 seconds)
5/29/2024, 2:37:07.969 PM"," DEBUG 2024-05-29 14:37:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:37:07.969 PM"," DEBUG 2024-05-29 14:37:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:39:07.968569-07:00 (in 119.999000 seconds)
5/29/2024, 2:39:07.969 PM"," DEBUG 2024-05-29 14:39:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:39:07.969 PM"," DEBUG 2024-05-29 14:39:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:39:07.971804-07:00 (in 0.002307 seconds)
5/29/2024, 2:39:07.972 PM"," DEBUG 2024-05-29 14:39:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:39:07.972 PM"," DEBUG 2024-05-29 14:39:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:41:07.968569-07:00 (in 119.996138 seconds)
5/29/2024, 2:41:07.969 PM"," DEBUG 2024-05-29 14:41:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:41:07.969 PM"," DEBUG 2024-05-29 14:41:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:43:07.968569-07:00 (in 119.999136 seconds)
5/29/2024, 2:43:07.969 PM"," DEBUG 2024-05-29 14:43:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:43:07.970 PM"," DEBUG 2024-05-29 14:43:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:45:07.968569-07:00 (in 119.998788 seconds)
5/29/2024, 2:45:07.969 PM"," DEBUG 2024-05-29 14:45:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:45:07.969 PM"," DEBUG 2024-05-29 14:45:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:47:07.968569-07:00 (in 119.999051 seconds)
5/29/2024, 2:47:07.969 PM"," DEBUG 2024-05-29 14:47:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:47:07.969 PM"," DEBUG 2024-05-29 14:47:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:47:07.971804-07:00 (in 0.002466 seconds)
5/29/2024, 2:47:07.973 PM"," DEBUG 2024-05-29 14:47:07,973 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:47:07.973 PM"," DEBUG 2024-05-29 14:47:07,973 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:49:07.968569-07:00 (in 119.994977 seconds)
5/29/2024, 2:49:07.969 PM"," DEBUG 2024-05-29 14:49:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:49:07.969 PM"," DEBUG 2024-05-29 14:49:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:51:07.968569-07:00 (in 119.999141 seconds)
5/29/2024, 2:51:07.969 PM"," DEBUG 2024-05-29 14:51:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:51:07.969 PM"," DEBUG 2024-05-29 14:51:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:53:07.968569-07:00 (in 119.999035 seconds)
5/29/2024, 2:53:07.969 PM"," DEBUG 2024-05-29 14:53:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:53:07.969 PM"," DEBUG 2024-05-29 14:53:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:55:07.968569-07:00 (in 119.999198 seconds)
5/29/2024, 2:55:07.969 PM"," DEBUG 2024-05-29 14:55:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:55:07.969 PM"," DEBUG 2024-05-29 14:55:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:55:07.971804-07:00 (in 0.002272 seconds)
5/29/2024, 2:55:07.972 PM"," DEBUG 2024-05-29 14:55:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:55:07.972 PM"," DEBUG 2024-05-29 14:55:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:57:07.968569-07:00 (in 119.996185 seconds)
5/29/2024, 2:57:07.969 PM"," DEBUG 2024-05-29 14:57:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:57:07.969 PM"," DEBUG 2024-05-29 14:57:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 07:59:07.968569-07:00 (in 119.999199 seconds)
5/29/2024, 2:59:07.969 PM"," DEBUG 2024-05-29 14:59:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 2:59:07.969 PM"," DEBUG 2024-05-29 14:59:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:01:07.968569-07:00 (in 119.999073 seconds)
5/29/2024, 3:01:07.969 PM"," DEBUG 2024-05-29 15:01:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:01:07.969 PM"," DEBUG 2024-05-29 15:01:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:03:07.968569-07:00 (in 119.999047 seconds)
5/29/2024, 3:03:07.969 PM"," DEBUG 2024-05-29 15:03:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:03:07.969 PM"," DEBUG 2024-05-29 15:03:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:03:07.970 PM"," DEBUG 2024-05-29 15:03:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:03:07.971804-07:00 (in 0.002240 seconds)
5/29/2024, 3:03:07.970 PM"," DEBUG 2024-05-29 15:03:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:03:07.971804-07:00 (in 0.002240 seconds)
5/29/2024, 3:03:07.972 PM"," DEBUG 2024-05-29 15:03:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:03:07.973 PM"," DEBUG 2024-05-29 15:03:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:05:07.968569-07:00 (in 119.996168 seconds)
5/29/2024, 3:05:07.969 PM"," DEBUG 2024-05-29 15:05:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:05:07.969 PM"," DEBUG 2024-05-29 15:05:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:07:07.968569-07:00 (in 119.999156 seconds)
5/29/2024, 3:07:07.970 PM"," DEBUG 2024-05-29 15:07:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:07:07.971 PM"," DEBUG 2024-05-29 15:07:07,971 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:09:07.968569-07:00 (in 119.997436 seconds)
5/29/2024, 3:09:07.970 PM"," DEBUG 2024-05-29 15:09:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:09:07.970 PM"," DEBUG 2024-05-29 15:09:07,970 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:11:07.968569-07:00 (in 119.998250 seconds)
5/29/2024, 3:11:07.969 PM"," DEBUG 2024-05-29 15:11:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:11:07.969 PM"," DEBUG 2024-05-29 15:11:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:11:07.971804-07:00 (in 0.002253 seconds)
5/29/2024, 3:11:07.972 PM"," DEBUG 2024-05-29 15:11:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:11:07.972 PM"," DEBUG 2024-05-29 15:11:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:13:07.968569-07:00 (in 119.996180 seconds)
5/29/2024, 3:13:07.969 PM"," DEBUG 2024-05-29 15:13:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:13:07.969 PM"," DEBUG 2024-05-29 15:13:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:15:07.968569-07:00 (in 119.999245 seconds)
5/29/2024, 3:15:07.969 PM"," DEBUG 2024-05-29 15:15:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:15:07.969 PM"," DEBUG 2024-05-29 15:15:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:17:07.968569-07:00 (in 119.999010 seconds)
5/29/2024, 3:17:07.969 PM"," DEBUG 2024-05-29 15:17:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:17:07.969 PM"," DEBUG 2024-05-29 15:17:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:19:07.968569-07:00 (in 119.999102 seconds)
5/29/2024, 3:19:07.969 PM"," DEBUG 2024-05-29 15:19:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:19:07.969 PM"," DEBUG 2024-05-29 15:19:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:19:07.971804-07:00 (in 0.002284 seconds)
5/29/2024, 3:19:07.972 PM"," DEBUG 2024-05-29 15:19:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:19:07.973 PM"," DEBUG 2024-05-29 15:19:07,973 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:21:07.968569-07:00 (in 119.994924 seconds)
5/29/2024, 3:21:07.969 PM"," DEBUG 2024-05-29 15:21:07,968 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:21:07.969 PM"," DEBUG 2024-05-29 15:21:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:23:07.968569-07:00 (in 119.999160 seconds)
5/29/2024, 3:23:07.969 PM"," DEBUG 2024-05-29 15:23:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:23:07.969 PM"," DEBUG 2024-05-29 15:23:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:25:07.968569-07:00 (in 119.999066 seconds)
5/29/2024, 3:25:07.969 PM"," DEBUG 2024-05-29 15:25:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:25:07.969 PM"," DEBUG 2024-05-29 15:25:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:27:07.968569-07:00 (in 119.999069 seconds)
5/29/2024, 3:27:07.969 PM"," DEBUG 2024-05-29 15:27:07,969 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:27:07.969 PM"," DEBUG 2024-05-29 15:27:07,969 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:27:07.971804-07:00 (in 0.002278 seconds)
5/29/2024, 3:27:07.972 PM"," DEBUG 2024-05-29 15:27:07,972 base 52 123216950421312 Looking for jobs to run
5/29/2024, 3:27:07.973 PM"," DEBUG 2024-05-29 15:27:07,972 base 52 123216950421312 Next wakeup is due at 2024-05-29 08:29:07.968569-07:00 (in 119.996174 seconds)
5/29/2024, 3:27:41.569 PM"," DEBUG 2024-05-29 15:27:41,569 base 52 128074064418624 Looking for jobs to run
5/29/2024, 3:27:41.569 PM"," DEBUG 2024-05-29 15:27:41,569 base 52 128074064418624 Next wakeup is due at 2024-05-29 08:29:41.562587-07:00 (in 119.993202 seconds)

from azure-sdk-for-python.

kashifkhan commented on June 17, 2024

Thanks @saadshaikh3 for the debug logs. Will take a look and reply back

from azure-sdk-for-python.

kashifkhan commented on June 17, 2024

@saadshaikh3 based on the current and previous logs, its still hard to decipher what exactly is going wrong here. There are two things to do as next steps:

enable frame level logging by doing the following client = ServiceBusClient(..., logging_enable=True)
open up an issue with the service team to see if something is going wrong there
how are you sending the ping? is it using the sdk or directly to the service? Could you include the code for that please

from azure-sdk-for-python.

github-actions commented on June 17, 2024

from azure-sdk-for-python.

saadshaikh3 commented on June 17, 2024

Hi @kashifkhan ,

The logging code that you specified I am unsure where should I use in my django application mainly because I am not using ServiceBusClient anywhere directly. I am just using service bus as a broker for celery.
I'll try if I can open an issue in Celery repo
Im sending a ping by scheduling a celery task every 8 mins using apscheduler.
Here's the code for task:

@shared_task(name="chat_app.tasks.ping",ignore_result=True)
def ping():
    print(f"running chat_app.tasks.ping")
    try:
        print(f"completed ping")
        return {"status":True,"message":"Ping successful. Worker consumer active"}
    except Exception as ex:
        logging.exception(ex)
        return {"status":False,"message":str(ex)}

def pingCelery():
    ping.delay()

The code for scheduling:

 scheduler.add_job(celery_bus_ping.pingCelery, 'interval', minutes=8, id="celery_bus_ping", replace_existing=True)

from azure-sdk-for-python.

kashifkhan commented on June 17, 2024

Hi @saadshaikh3 , I didnt realize this was not being used directly. It might be best to see if celery can provide more information on the usage. The other thing would be to reach out to the Service Bus team and see if they can see on their side if something is going on and can get more information.

As for the ping, its not pinging the actual Service Bus service. In order to ping you need to use a send operation on the client to send a dummy message

from azure-sdk-for-python.

[ServiceBus] Unrecoverable error: "ServiceBusConnectionError('Link detached unexpectedly. Error condition: amqp:unknown-error.')" about azure-sdk-for-python HOT 14 OPEN

Comments (14)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent