aws-samples / amazon-a2i-sample-jupyter-notebooks Goto Github PK
View Code? Open in Web Editor NEWSample Jupyter Notebooks for Amazon Augmented AI (A2I)
Home Page: https://aws.amazon.com/augmented-ai/
License: Apache License 2.0
Sample Jupyter Notebooks for Amazon Augmented AI (A2I)
Home Page: https://aws.amazon.com/augmented-ai/
License: Apache License 2.0
Hi there,
I really appreciate the notebook Amazon A2I with Amazon SageMaker for object detection and model retraining to know how to use A2I and incremental training.
I have a question about how to retrain with new labeled data. In the current code, you use only new dataset, but the guide of the incremental training says that we should use "an expanded dataset". Does this mean that we have to use the datasets including both one from the first training and one obtained from A2I? Or can we use only the new dataset by using an augmented manifest file?
At least, when I retrained my model with only new dataset using RecordIO format, the model accuracy highly decreased.
I hope you would modify the current code to use "an expanded dataset", which should be more helpful for us to understand how to use A2I and incremental training.
Thanks,
It's a bit unclear how to run this locally, but managed to bypass errors around credentials not being found by passing a boto3.Session to the sagemaker.Session that calls one of my "profile_name"s.
However, I'm getting this error everytime I try to make a prediction. Specifically when I run
object_detector = sagemaker.predictor.RealTimePredictor(endpoint=endpoint_name,sagemaker_session=sess)
with open(test_photos[2], 'rb') as image:
f = image.read()
b = bytearray(f)
results = object_detector.predict(b)
I get
---------------------------------------------------------------------------
BrokenPipeError Traceback (most recent call last)
~/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
599 body=body, headers=headers,
--> 600 chunked=chunked)
601
~/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
353 else:
--> 354 conn.request(method, url, **httplib_request_kw)
355
~/anaconda3/lib/python3.7/http/client.py in request(self, method, url, body, headers, encode_chunked)
1228 """Send a complete request to the server."""
-> 1229 self._send_request(method, url, body, headers, encode_chunked)
1230
~/anaconda3/lib/python3.7/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
91 rval = super(AWSConnection, self)._send_request(
---> 92 method, url, body, headers, *args, **kwargs)
93 self._expect_header_set = False
~/anaconda3/lib/python3.7/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
1274 body = _encode(body, 'body')
-> 1275 self.endheaders(body, encode_chunked=encode_chunked)
1276
~/anaconda3/lib/python3.7/http/client.py in endheaders(self, message_body, encode_chunked)
1223 raise CannotSendHeader()
-> 1224 self._send_output(message_body, encode_chunked=encode_chunked)
1225
~/anaconda3/lib/python3.7/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
142 # we must run the risk of Nagle.
--> 143 self.send(message_body)
144
~/anaconda3/lib/python3.7/site-packages/botocore/awsrequest.py in send(self, str)
202 return
--> 203 return super(AWSConnection, self).send(str)
204
~/anaconda3/lib/python3.7/http/client.py in send(self, data)
976 try:
--> 977 self.sock.sendall(data)
978 except TypeError:
~/anaconda3/lib/python3.7/ssl.py in sendall(self, data, flags)
1014 while count < amount:
-> 1015 v = self.send(byte_view[count:])
1016 count += v
~/anaconda3/lib/python3.7/ssl.py in send(self, data, flags)
983 self.__class__)
--> 984 return self._sslobj.write(data)
985 else:
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
ProtocolError Traceback (most recent call last)
~/anaconda3/lib/python3.7/site-packages/botocore/httpsession.py in send(self, request)
262 decode_content=False,
--> 263 chunked=self._chunked(request.headers),
264 )
~/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
637 retries = retries.increment(method, url, error=e, _pool=self,
--> 638 _stacktrace=sys.exc_info()[2])
639 retries.sleep()
~/anaconda3/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
343 # Disabled, indicate to re-raise the error.
--> 344 raise six.reraise(type(error), error, _stacktrace)
345
~/anaconda3/lib/python3.7/site-packages/urllib3/packages/six.py in reraise(tp, value, tb)
684 if value.__traceback__ is not tb:
--> 685 raise value.with_traceback(tb)
686 raise value
~/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
599 body=body, headers=headers,
--> 600 chunked=chunked)
601
~/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
353 else:
--> 354 conn.request(method, url, **httplib_request_kw)
355
~/anaconda3/lib/python3.7/http/client.py in request(self, method, url, body, headers, encode_chunked)
1228 """Send a complete request to the server."""
-> 1229 self._send_request(method, url, body, headers, encode_chunked)
1230
~/anaconda3/lib/python3.7/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
91 rval = super(AWSConnection, self)._send_request(
---> 92 method, url, body, headers, *args, **kwargs)
93 self._expect_header_set = False
~/anaconda3/lib/python3.7/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
1274 body = _encode(body, 'body')
-> 1275 self.endheaders(body, encode_chunked=encode_chunked)
1276
~/anaconda3/lib/python3.7/http/client.py in endheaders(self, message_body, encode_chunked)
1223 raise CannotSendHeader()
-> 1224 self._send_output(message_body, encode_chunked=encode_chunked)
1225
~/anaconda3/lib/python3.7/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
142 # we must run the risk of Nagle.
--> 143 self.send(message_body)
144
~/anaconda3/lib/python3.7/site-packages/botocore/awsrequest.py in send(self, str)
202 return
--> 203 return super(AWSConnection, self).send(str)
204
~/anaconda3/lib/python3.7/http/client.py in send(self, data)
976 try:
--> 977 self.sock.sendall(data)
978 except TypeError:
~/anaconda3/lib/python3.7/ssl.py in sendall(self, data, flags)
1014 while count < amount:
-> 1015 v = self.send(byte_view[count:])
1016 count += v
~/anaconda3/lib/python3.7/ssl.py in send(self, data, flags)
983 self.__class__)
--> 984 return self._sslobj.write(data)
985 else:
ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
During handling of the above exception, another exception occurred:
ConnectionClosedError Traceback (most recent call last)
<ipython-input-13-ea07fcaa5d16> in <module>
3 f = image.read()
4 b = bytearray(f)
----> 5 results = object_detector.predict(b,)
~/anaconda3/lib/python3.7/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model)
108
109 request_args = self._create_request_args(data, initial_args, target_model)
--> 110 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
111 return self._handle_response(response)
112
~/anaconda3/lib/python3.7/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
314 "%s() only accepts keyword arguments." % py_operation_name)
315 # The "self" in this scope is referring to the BaseClient.
--> 316 return self._make_api_call(operation_name, kwargs)
317
318 _api_call.__name__ = str(py_operation_name)
~/anaconda3/lib/python3.7/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
620 else:
621 http, parsed_response = self._make_request(
--> 622 operation_model, request_dict, request_context)
623
624 self.meta.events.emit(
~/anaconda3/lib/python3.7/site-packages/botocore/client.py in _make_request(self, operation_model, request_dict, request_context)
639 def _make_request(self, operation_model, request_dict, request_context):
640 try:
--> 641 return self._endpoint.make_request(operation_model, request_dict)
642 except Exception as e:
643 self.meta.events.emit(
~/anaconda3/lib/python3.7/site-packages/botocore/endpoint.py in make_request(self, operation_model, request_dict)
100 logger.debug("Making request for %s with params: %s",
101 operation_model, request_dict)
--> 102 return self._send_request(request_dict, operation_model)
103
104 def create_request(self, params, operation_model=None):
~/anaconda3/lib/python3.7/site-packages/botocore/endpoint.py in _send_request(self, request_dict, operation_model)
135 request, operation_model, context)
136 while self._needs_retry(attempts, operation_model, request_dict,
--> 137 success_response, exception):
138 attempts += 1
139 # If there is a stream associated with the request, we need
~/anaconda3/lib/python3.7/site-packages/botocore/endpoint.py in _needs_retry(self, attempts, operation_model, request_dict, response, caught_exception)
254 event_name, response=response, endpoint=self,
255 operation=operation_model, attempts=attempts,
--> 256 caught_exception=caught_exception, request_dict=request_dict)
257 handler_response = first_non_none_response(responses)
258 if handler_response is None:
~/anaconda3/lib/python3.7/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
354 def emit(self, event_name, **kwargs):
355 aliased_event_name = self._alias_event_name(event_name)
--> 356 return self._emitter.emit(aliased_event_name, **kwargs)
357
358 def emit_until_response(self, event_name, **kwargs):
~/anaconda3/lib/python3.7/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
226 handlers.
227 """
--> 228 return self._emit(event_name, kwargs)
229
230 def emit_until_response(self, event_name, **kwargs):
~/anaconda3/lib/python3.7/site-packages/botocore/hooks.py in _emit(self, event_name, kwargs, stop_on_response)
209 for handler in handlers_to_call:
210 logger.debug('Event %s: calling handler %s', event_name, handler)
--> 211 response = handler(**kwargs)
212 responses.append((handler, response))
213 if stop_on_response and response is not None:
~/anaconda3/lib/python3.7/site-packages/botocore/retryhandler.py in __call__(self, attempts, response, caught_exception, **kwargs)
181
182 """
--> 183 if self._checker(attempts, response, caught_exception):
184 result = self._action(attempts=attempts)
185 logger.debug("Retry needed, action of: %s", result)
~/anaconda3/lib/python3.7/site-packages/botocore/retryhandler.py in __call__(self, attempt_number, response, caught_exception)
249 def __call__(self, attempt_number, response, caught_exception):
250 should_retry = self._should_retry(attempt_number, response,
--> 251 caught_exception)
252 if should_retry:
253 if attempt_number >= self._max_attempts:
~/anaconda3/lib/python3.7/site-packages/botocore/retryhandler.py in _should_retry(self, attempt_number, response, caught_exception)
275 # If we've exceeded the max attempts we just let the exception
276 # propogate if one has occurred.
--> 277 return self._checker(attempt_number, response, caught_exception)
278
279
~/anaconda3/lib/python3.7/site-packages/botocore/retryhandler.py in __call__(self, attempt_number, response, caught_exception)
315 for checker in self._checkers:
316 checker_response = checker(attempt_number, response,
--> 317 caught_exception)
318 if checker_response:
319 return checker_response
~/anaconda3/lib/python3.7/site-packages/botocore/retryhandler.py in __call__(self, attempt_number, response, caught_exception)
221 elif caught_exception is not None:
222 return self._check_caught_exception(
--> 223 attempt_number, caught_exception)
224 else:
225 raise ValueError("Both response and caught_exception are None.")
~/anaconda3/lib/python3.7/site-packages/botocore/retryhandler.py in _check_caught_exception(self, attempt_number, caught_exception)
357 # the MaxAttemptsDecorator is not interested in retrying the exception
358 # then this exception just propogates out past the retry code.
--> 359 raise caught_exception
~/anaconda3/lib/python3.7/site-packages/botocore/endpoint.py in _do_get_response(self, request, operation_model)
198 http_response = first_non_none_response(responses)
199 if http_response is None:
--> 200 http_response = self._send(request)
201 except HTTPClientError as e:
202 return (None, e)
~/anaconda3/lib/python3.7/site-packages/botocore/endpoint.py in _send(self, request)
267
268 def _send(self, request):
--> 269 return self.http_session.send(request)
270
271
~/anaconda3/lib/python3.7/site-packages/botocore/httpsession.py in send(self, request)
292 error=e,
293 request=request,
--> 294 endpoint_url=request.url
295 )
296 except Exception as e:
ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://runtime.sagemaker.us-east-1.amazonaws.com/endpoints/DEMO-object-detection-augmented-ai-2020-05-19-00-18-57/invocations".
If I run
results = object_detector.predict("dummy text")
I get something I can understand and expect:
---------------------------------------------------------------------------
ModelError Traceback (most recent call last)
<ipython-input-22-dd7ea88a4b12> in <module>
3 f = image.read()
4 b = bytearray(f)
----> 5 results = object_detector.predict("dummy tex")
~/anaconda3/lib/python3.7/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model)
108
109 request_args = self._create_request_args(data, initial_args, target_model)
--> 110 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
111 return self._handle_response(response)
112
~/anaconda3/lib/python3.7/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
314 "%s() only accepts keyword arguments." % py_operation_name)
315 # The "self" in this scope is referring to the BaseClient.
--> 316 return self._make_api_call(operation_name, kwargs)
317
318 _api_call.__name__ = str(py_operation_name)
~/anaconda3/lib/python3.7/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
633 error_code = parsed_response.get("Error", {}).get("Code")
634 error_class = self.exceptions.from_code(error_code)
--> 635 raise error_class(parsed_response, operation_name)
636 else:
637 return parsed_response
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "unable to evaluate payload provided". See https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/DEMO-object-detection-augmented-ai-2020-05-19-00-18-57 in account 488507749156 for more information.
Name: boto3
Version: 1.13.12
Name: sagemaker
Version: 1.58.2
Name: botocore
Version: 1.16.12
Python 3.7.3
in notebook of Amazon Augmented AI (A2I) and Textract AnalyzeDocument
when run
uniqueId = str(uuid.uuid4())
print(uniqueId)
flowDefinitionName = f'fd-textract-{uniqueId}'
print (flowDefinitionName)
flowDefinitionArn = create_flow_definition(flowDefinitionName)
print (flowDefinitionArn)
The error happened,
ClientError: An error occurred (ValidationException) when calling the CreateFlowDefinition operation: 1 validation error detected: Value 'arn:aws:sagemaker:ap-southeast-1:xxxxxxxxxxxx:flow-definition/a2idemotextract' at 'humanLoopConfig.workteamArn' failed to satisfy constraint: Member must satisfy regular expression pattern: arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:workteam/.*
Is there any problem?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.