xadrianzetx / coral-deeplab Goto Github PK

Hi @xadrianzetx, thanks again for this repository and supporting it. I am trying to create a custom model using the code in the Jupyter notebook: https://colab.research.google.com/drive/1eJFOg_1o5D-osHZseNlpE-WGR-p8jDqa?usp=sharing

It simply creates a CoralDeepLabV3 model and compiles it for the Edge TPU. However, when running it with the inference script the result is not as expected:

Any idea what I could be doing wrong? Is it required to use an older version of the edgetpu-compiler if the newest version has no errors?

Pasting the code here as well:

pip install git+https://github.com/xadrianzetx/coral-deeplab.git

! curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
! echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
! sudo apt-get update
! sudo apt-get install edgetpu-compiler	
! edgetpu_compiler -v

import os
import re
import sys
import uuid
import unittest
import subprocess

import numpy as np
import tensorflow as tf

import coral_deeplab as cdl

def fake_dataset_generator(shape, n_iter):
    def dataset():
        for _ in range(n_iter):
            data = np.random.randn(*shape)
            data *= (1 / 255)
            batch = np.expand_dims(data, axis=0)
            yield [batch.astype(np.float32)]
    return dataset


def quantize_and_compile(model, dataset):

    # setup tflite converter
    converter = tf.lite.TFLiteConverter.from_keras_model(model)
    converter.representative_dataset = dataset
    converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
    converter.inference_input_type = tf.uint8
    converter.inference_output_type = tf.uint8
    converter.optimizations = [tf.lite.Optimize.DEFAULT]

    # quantize
    quantized = converter.convert()
    model_name = f'{uuid.uuid4()}.tflite'
    model_path = os.path.join(os.getcwd(), model_name)
    open(model_path, 'wb').write(quantized)

    # compile
    cmd = ['edgetpu_compiler', '-a', '-s', model_path]
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    p.wait()
    stdout = p.stdout.read()

    return stdout.decode()

model = cdl.applications.CoralDeepLabV3(weights='pascal_voc')
datagen = fake_dataset_generator((513, 513, 3), 10)
stdout = quantize_and_compile(model, datagen)
print(stdout)

! cp /content/10582856-eda3-4088-bf25-1e6a7e5e75de_edgetpu.tflite /content/drive/MyDrive/

Hello!
It works more or less, thank you! Is it possible to change NN input size to 1600x800?

https://github.com/xadrianzetx/coral-deeplab/actions/runs/2896118781

https://peps.python.org/pep-0518/

Thank you for providing this repo. I think I was able to compile a model for the coral successfully, and I tried to use the segmentation inference code provided by google: https://github.com/google-coral/pycoral/blob/master/examples/semantic_segmentation.py
However the output from that inference did not make much sense, especially compared to the output from using one of the models provided by google.

Model from google:

Compiled model:

Is there a reason this inference script will not work with models from this repo? Would you happen to have an inference script that would work better?

Hello,
I am using https://github.com/srihari-humbarwadi/DeepLabV3_Plus-Tensorflow2.0 trained with my dataset. I am going to migrate from GPU to TPU. Thank you for your project!
However I have problem to install it. Please, help to solve problem described hereunder.

Hardware: asus tinker edge t
OS: Tinker_Edge_T-Mendel-Eagle-V3.0.2-20201015
Python: Python 3.7.3

Issue:
I run
pip3 install git+https://github.com/xadrianzetx/coral-deeplab.git

Collecting git+https://github.com/xadrianzetx/coral-deeplab.git
Cloning https://github.com/xadrianzetx/coral-deeplab.git to /tmp/pip-req-build-12yqg40c
Collecting numpy>=1.19.5 (from coral-deeplab==0.2)
Downloading https://files.pythonhosted.org/packages/66/03/818876390c7ff4484d5a05398a618cfdaf0a2b9abb3a7c7ccd59fe181008/numpy-1.21.0.zip (10.3MB)
100% |████████████████████████████████| 10.3MB 40kB/s
Installing build dependencies ... done
Collecting tensorflow-gpu==2.4.0 (from coral-deeplab==0.2)
Could not find a version that satisfies the requirement tensorflow-gpu==2.4.0 (from coral-deeplab==0.2) (from versions: )
No matching distribution found for tensorflow-gpu==2.4.0 (from coral-deeplab==0.2)

Would it be possible to update this for Python 3.8? Google Colab has stopped supporting 3.7 since it is end of life in a few months

Can you please help on compiling and loss plus metrics? That would great.

Thanks,
Steve

Hello, I have tried to train from scratch as you advised However there is problem, and I think it is because of model structure. Please, kind to help me clarify reason.

Comparing to code for GPU there are differences:

• Packed in docker under dockercompose
• Tensorflow 2.0 replaced by TF 2.4.0
• Python 3.6 changed to 3.7

Here is docker:

FROM tensorflow/tensorflow:2.4.2-gpu

RUN add-apt-repository -y --remove ppa:fkrull/deadsnakes && apt-get -y update && apt-get remove -y --purge python3.6

WORKDIR /workspace

RUN DEBIAN_FRONTEND="noninteractive" apt-get -y install tzdata && apt-get install -y python3-opencv git python3.7 nano mc

RUN cd /usr/local/bin && unlink python && ln -s /usr/bin/python3.7 python
RUN cd /usr/bin && unlink python3 && ln -s /usr/bin/python3.7 python3

RUN apt-get purge -y python-pip python3-pip && apt-get install -y python-pip python3-pip
RUN python3 -m pip install --user --upgrade pip
RUN pip3 install requests numpy>=1.19.5
RUN pip3 install git+https://github.com/xadrianzetx/coral-deeplab.git

Last command forces to TF2.4.0. FROM tensorflow/tensorflow:2.4.2-gpu is to be sure in GPU support in docker

Compose

version: "3.5"

services:
tpu:
container_name: tpu
restart: always
build: .
volumes:
- .:/workspace
- ../prepare_dataset:/dataset
network_mode: host
command: /bin/bash #python3 service.py
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]

train.py

from glob import glob
import tensorflow as tf
from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint
import coral_deeplab as cdl

print('TensorFlow', tf.version)

batch_size = 6 # 8 due to memory it was is changed 6
H, W = 512, 512

num_classes = 2 # 6 # 7 # 34 (count is 0th ground + yours Thus if u have 1 class than put 2)

image_list = sorted(glob('/dataset/train_images/.png'))
mask_list = sorted(glob('/dataset/train_masks/_gtFine_labelIds*.png'))

val_image_list = sorted(glob('/dataset/val_images/'))
val_mask_list = sorted(glob('/dataset/val_masks/_gtFine_labelIds*'))

print('Found', len(image_list), 'training images')
print('Found', len(val_image_list), 'validation images')

for i in range(len(image_list)):
assert image_list[i].split('/')[-1].split('_leftImg8bit')[0] == mask_list[i].split('/')[-1].split('_gtFine_labelIds')[0]

for i in range(len(val_image_list)):
assert val_image_list[i].split('/')[-1].split('_leftImg8bit')[0] == val_mask_list[i].split('/')[-1].split('_gtFine_labelIds')[0]

#id_to_color = {0: (0, 0, 0), 1: (0, 0, 0), 2: (0, 0, 0), 3: (0, 0, 0), 4: (0, 0, 0), 5: (111, 74, 0), 6: (81, 0, 81), 7: (128, 64, 128), 8: (244, 35, 232), 9: (250, 170, 160), 10: (230, 150, 140), 11: (70, 70, 70), 12: (102, 102, 156), 13: (190, 153, 153), 14: (180, 165, 180), 15: (150, 100, 100), 16: (150, 120, 90), 17: (153, 153, 153), 18: (153, 153, 153), 19: (250, 170, 30), 20: (220, 220, 0), 21: (107, 142, 35), 22: (152, 251, 152), 23: (70, 130, 180), 24: (220, 20, 60), 25: (255, 0, 0), 26: (0, 0, 142), 27: (0, 0, 70), 28: (0, 60, 100), 29: (0, 0, 90), 30: (0, 0, 110), 31: (0, 80, 100), 32: (0, 0, 230), 33: (119, 11, 32), -1: (0, 0, 142)}

def get_image(image_path, img_height=800, img_width=1600, mask=False, flip=0):
img = tf.io.read_file(image_path)
if not mask:
img = tf.cast(tf.image.decode_png(img, channels=3), dtype=tf.float32)
img = tf.image.resize(images=img, size=[img_height, img_width])

    #Do not remove! 
    img = img[:, :, ::-1] - tf.constant([103.939, 116.779, 123.68])
else:
    img = tf.image.decode_png(img, channels=1)
    img = tf.cast(tf.image.resize(images=img, size=[
                  img_height, img_width]), dtype=tf.uint8)

return img

def random_crop(image, mask, H=512, W=512):
image_dims = image.shape
offset_h = tf.random.uniform(
shape=(1,), maxval=image_dims[0] - H, dtype=tf.int32)[0]
offset_w = tf.random.uniform(
shape=(1,), maxval=image_dims[1] - W, dtype=tf.int32)[0]

image = tf.image.crop_to_bounding_box(image,
                                      offset_height=offset_h,
                                      offset_width=offset_w,
                                      target_height=H,
                                      target_width=W)
mask = tf.image.crop_to_bounding_box(mask,
                                     offset_height=offset_h,
                                     offset_width=offset_w,
                                     target_height=H,
                                     target_width=W)
return image, mask

def load_data(image_path, mask_path, H=512, W=512):
flip = tf.random.uniform(shape=[1, ], minval=0, maxval=2, dtype=tf.int32)[0]
image, mask = get_image(image_path, flip=flip), get_image(mask_path, mask=True, flip=flip)
image, mask = random_crop(image, mask, H=H, W=W)
return image, mask

train_dataset = tf.data.Dataset.from_tensor_slices((image_list, mask_list))

train_dataset = train_dataset.shuffle(buffer_size=128)
train_dataset = train_dataset.apply(
tf.data.experimental.map_and_batch(map_func=load_data,
batch_size=batch_size,
num_parallel_calls=tf.data.experimental.AUTOTUNE,
drop_remainder=True))

train_dataset = train_dataset.repeat()
train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)
print(train_dataset)

val_dataset = tf.data.Dataset.from_tensor_slices((val_image_list,
val_mask_list))
val_dataset = val_dataset.apply(
tf.data.experimental.map_and_batch(map_func=load_data,
batch_size=batch_size,
num_parallel_calls=tf.data.experimental.AUTOTUNE,
drop_remainder=True))
val_dataset = val_dataset.repeat()
val_dataset = val_dataset.prefetch(tf.data.experimental.AUTOTUNE)

loss = tf.losses.SparseCategoricalCrossentropy(from_logits=True)

strategy = tf.distribute.MirroredStrategy()

with strategy.scope():
model = cdl.applications.CoralDeepLabV3(input_shape=(W, H, 3))
isinstance(model, tf.keras.Model)

for layer in model.layers:
    if isinstance(layer, tf.keras.layers.BatchNormalization):
        layer.momentum = 0.9997
        layer.epsilon = 1e-5
    elif isinstance(layer, tf.keras.layers.Conv2D):
        layer.kernel_regularizer = tf.keras.regularizers.l2(1e-4)

model.compile(loss=loss,
              optimizer=tf.optimizers.Adam(learning_rate=1e-4),
              metrics=['accuracy'])

#model.save("deeplab.h5")

tb = TensorBoard(log_dir='logs', write_graph=True, update_freq='batch')
mc = ModelCheckpoint(mode='min', filepath='top_weights.h5',
monitor='val_loss', save_best_only='True',
save_weights_only='True', verbose=1)
callbacks = [mc, tb]

model.fit(train_dataset,
steps_per_epoch=len(image_list) // batch_size,
epochs=150,
validation_data=val_dataset,
validation_steps=len(val_image_list) // batch_size,
callbacks=callbacks)

sudo docker-compose run tpu bash will start container

Output is

user@pc:/workspace# python train.py
2021-07-09 20:27:11.197491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
TensorFlow 2.4.0
Found 628 training images
Found 62 validation images

2021-07-09 20:27:12.032295: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-07-09 20:27:12.032810: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-07-09 20:27:12.275391: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.276037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6705GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-07-09 20:27:12.276054: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-07-09 20:27:12.277638: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-07-09 20:27:12.277666: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-07-09 20:27:12.278321: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-09 20:27:12.278467: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-09 20:27:12.280193: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-09 20:27:12.280575: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-07-09 20:27:12.280682: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-09 20:27:12.280754: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.281367: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.281801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-09 20:27:12.282319: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-07-09 20:27:12.282396: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.282840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6705GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s
2021-07-09 20:27:12.282857: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-07-09 20:27:12.282868: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-07-09 20:27:12.282878: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-07-09 20:27:12.282888: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-09 20:27:12.282898: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-09 20:27:12.282911: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-09 20:27:12.282926: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-07-09 20:27:12.282940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-09 20:27:12.282988: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.283568: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.283984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-09 20:27:12.284006: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-07-09 20:27:12.626301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-09 20:27:12.626336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-07-09 20:27:12.626344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-07-09 20:27:12.626466: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.626825: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.627144: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-09 20:27:12.627447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10271 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
WARNING:tensorflow:From train.py:88: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map(map_func, num_parallel_calls) followed by tf.data.Dataset.batch(batch_size, drop_remainder). Static tf.data optimizations will take care of using the fused implementation.
<PrefetchDataset shapes: ((6, 512, 512, 3), (6, 512, 512, 1)), types: (tf.float32, tf.uint8)>
2021-07-09 20:27:14.102904: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2021-07-09 20:27:14.102937: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2021-07-09 20:27:14.102974: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2021-07-09 20:27:14.103482: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcupti.so.11.0
2021-07-09 20:27:14.213655: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2021-07-09 20:27:14.213828: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] CUPTI activity buffer flushed
2021-07-09 20:27:14.387837: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:656] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Found an unshardable source dataset: name: "TensorSliceDataset/_2"
op: "TensorSliceDataset"
input: "Placeholder/_0"
input: "Placeholder/_1"
attr {
key: "Toutput_types"
value {
list {
type: DT_STRING
type: DT_STRING
}
}
}
attr {
key: "output_shapes"
value {
list {
shape {
}
shape {
}
}
}
}

2021-07-09 20:27:14.495200: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-07-09 20:27:14.499718: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 4200000000 Hz
Epoch 1/150
Traceback (most recent call last):
File "train.py", line 140, in
callbacks=callbacks)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
tmp_logs = self.train_function(iterator)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 828, in call
result = self._call(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 871, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 726, in _initialize
*args, **kwds))
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 2969, in _get_concrete_function_internal_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3361, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 3206, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py", line 990, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/def_function.py", line 634, in wrapped_fn
out = weak_wrapped_fn().wrapped(*args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py", line 977, in wrapper
raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:805 train_function  *
    return step_function(self, iterator)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:795 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:1259 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:2730 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/mirrored_strategy.py:629 _call_for_each_replica
    self._container_strategy(), fn, args, kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/mirrored_run.py:93 call_for_each_replica
    return _call_for_each_replica(strategy, fn, args, kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/mirrored_run.py:234 _call_for_each_replica
    coord.join(threads)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/coordinator.py:389 join
    six.reraise(*self._exc_info_to_raise)
/usr/local/lib/python3.7/dist-packages/six.py:703 reraise
    raise value
/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/coordinator.py:297 stop_on_exception
    yield
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/mirrored_run.py:323 run
    self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:788 run_step  **
    outputs = model.train_step(data)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:756 train_step
    y, y_pred, sample_weight, regularization_losses=self.losses)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/compile_utils.py:203 __call__
    loss_value = loss_obj(y_t, y_p, sample_weight=sw)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/losses.py:152 __call__
    losses = call_fn(y_true, y_pred)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/losses.py:256 call  **
    return ag_fn(y_true, y_pred, **self._fn_kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
    return target(*args, **kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/losses.py:1569 sparse_categorical_crossentropy
    y_true, y_pred, from_logits=from_logits, axis=axis)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
    return target(*args, **kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/backend.py:4941 sparse_categorical_crossentropy
    labels=target, logits=output)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
    return target(*args, **kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/nn_ops.py:4241 sparse_softmax_cross_entropy_with_logits_v2
    labels=labels, logits=logits, name=name)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
    return target(*args, **kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/nn_ops.py:4156 sparse_softmax_cross_entropy_with_logits
    logits.get_shape()))

ValueError: Shape mismatch: The shape of labels (received (1572864,)) should equal the shape of logits except for the last dimension (received (6144, 30)).