Code Monkey home page Code Monkey logo

Comments (14)

mwalenia avatar mwalenia commented on September 21, 2024

@zhitaoli do you have any ideas as to why this happened?

from tfx.

zhitaoli avatar zhitaoli commented on September 21, 2024

from tfx.

gogasca avatar gogasca commented on September 21, 2024

I got the same error using the setup.py libraries:

import setuptools

# LINT.IfChange
TF_VERSION = '1.12.0'
# LINT.ThenChange(train_mlengine.sh, start_model_server_mlengine.sh)

# LINT.IfChange
BEAM_VERSION = '2.11.0'
# LINT.ThenChange(setup_beam_on_flink.sh)

if __name__ == '__main__':
  setuptools.setup(
      name='tfx_chicago_taxi',
      version='0.12.0',
      packages=setuptools.find_packages(),
      install_requires=[
          'apache-beam[gcp]==' + BEAM_VERSION,
          'jupyter==1.0',
          'numpy==1.14.5',
          'protobuf==3.6.1',
          'tensorflow==' + TF_VERSION,
          'tensorflow-data-validation==0.12.0',
          'tensorflow-metadata==0.12.1',
          'tensorflow-model-analysis==0.12.1',
          'tensorflow-serving-api==1.12.0',
          'tensorflow-transform==0.12.0',
      ],
      python_requires='>=2.7,<3')
Traceback (most recent call last):
  File "tfdv_analyze_and_validate.py", line 191, in <module>
    main()
  File "tfdv_analyze_and_validate.py", line 176, in main
    pipeline_args=pipeline_args)
  File "tfdv_analyze_and_validate.py", line 121, in compute_stats
    statistics_pb2.DatasetFeatureStatisticsList)))
  File "/Users/gogasca/Documents/Development/dpe/tfx_taxi_cloud/taxi/lib/python2.7/site-packages/apache_beam/pipeline.py", line 426, in __exit__
    self.run().wait_until_finish()
  File "/Users/gogasca/Documents/Development/dpe/tfx_taxi_cloud/taxi/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1238, in wait_until_finish
    (self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 773, in run
    self._load_main_session(self.local_staging_directory)
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 489, in _load_main_session
    pickler.load_session(session_file)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 269, in load_session
    return dill.load_session(file_path)
  File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 410, in load_session
    module = unpickler.load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 828, in _import_module
    return getattr(__import__(module, None, None, [obj]), obj)
  File "/usr/local/lib/python2.7/dist-packages/trainer/taxi.py", line 19, in <module>
    from tensorflow_transform import coders as tft_coders
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/__init__.py", line 19, in <module>
    from tensorflow_transform.analyzers import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/analyzers.py", line 39, in <module>
    from tensorflow_transform import tf_utils
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/tf_utils.py", line 24, in <module>
    from tensorflow.contrib.proto.python.ops import encode_proto_op
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/__init__.py", line 48, in <module>
    from tensorflow.contrib import distribute
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/distribute/__init__.py", line 34, in <module>
    from tensorflow.contrib.distribute.python.tpu_strategy import TPUStrategy
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/distribute/python/tpu_strategy.py", line 27, in <module>
    from tensorflow.contrib.tpu.python.ops import tpu_ops
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/__init__.py", line 73, in <module>
    from tensorflow.contrib.tpu.python.tpu.keras_support import tpu_model as keras_to_tpu_model
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/keras_support.py", line 71, in <module>
    from tensorflow.python.estimator import model_fn as model_fn_lib
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/__init__.py", line 25, in <module>
    import tensorflow.python.estimator.estimator_lib
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator_lib.py", line 22, in <module>
    from tensorflow.python.estimator.canned.baseline import BaselineClassifier
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/canned/baseline.py", line 50, in <module>
    from tensorflow.python.estimator import estimator
ImportError: cannot import name estimator

This is when running latest tfdv_analyze_and_validate_dataflow.sh

from tfx.

mwalenia avatar mwalenia commented on September 21, 2024

@zhitaoli I'm using the github master branch version. As for the pipeline, I'm using the one constructed in tfdv_analyze_and_validate_dataflow.sh and running the file with the command:

python tfdv_analyze_and_validate.py \
  --input bigquery-public-data.chicago_taxi_trips.taxi_trips \
  --infer_schema \
  --stats_path ${TFDV_OUTPUT_PATH}/train_stats.tfrecord \
  --schema_path ${SCHEMA_PATH} \
  --project ${MYPROJECT} \
  --region us-central1 \
  --temp_location ${TEMP_PATH} \
  --experiments shuffle_mode=auto \
  --job_name ${JOB_ID} \
  --setup_file ./setup.py \
  --save_main_session \
  --runner DataflowRunner \
  --max_rows=${MAX_ROWS}

from tfx.

gogasca avatar gogasca commented on September 21, 2024

Using:

import setuptools

# LINT.IfChange
TF_VERSION = '1.13.1'
# LINT.ThenChange(train_mlengine.sh, start_model_server_mlengine.sh)

# LINT.IfChange
BEAM_VERSION = '2.12.0'
# LINT.ThenChange(setup_beam_on_flink.sh)

if __name__ == '__main__':
  setuptools.setup(
      name='tfx_chicago_taxi',
      version='0.13.0',
      packages=setuptools.find_packages(),
      install_requires=[
          'apache-beam[gcp]==' + BEAM_VERSION,
          'jupyter==1.0',
          'protobuf==3.7.1',
          'tensorflow==' + TF_VERSION,
          'tensorflow-data-validation==0.13.1',
          'tensorflow-metadata==0.13.0',
          'tensorflow-model-analysis==0.13.2',
          'tensorflow-serving-api==1.13.0',
          'tensorflow-transform==0.13.0',
      ],
      python_requires='>=2.7,<3')

In a Python 2.7 virtualenv seems to be working now

from tfx.

mwalenia avatar mwalenia commented on September 21, 2024

@gogasca It worked for me too, thanks! What do we do now? Do you want to submit a PR updating the dependencies for chicago_taxi, or should I do it?

Once again, great find

from tfx.

zhitaoli avatar zhitaoli commented on September 21, 2024

There should be a submit from @ruoyu90 soon which will fix this. Thanks for the help.

from tfx.

gogasca avatar gogasca commented on September 21, 2024

This PR seems to include changes as well #91

from tfx.

zhitaoli avatar zhitaoli commented on September 21, 2024

This PR seems to include changes as well #91

Yes this is the PR I mentioned. Waiting for @ruoyu90 to submit.

from tfx.

gogasca avatar gogasca commented on September 21, 2024

Thanks, just started training and looks like just some minor updates in chicago_taxi_client.py, task.py and model.py needed in order to avoid import errors.

from tfx.

mwalenia avatar mwalenia commented on September 21, 2024

@ruoyu90 It is necessary to limit the version of scikit-learn to 0.20 in setup.py of the chicago_taxi (and probably in more places), as version 0.21.0 requires python 3.5 since today
source

from tfx.

mwalenia avatar mwalenia commented on September 21, 2024

@zhitaoli I also seem to have a problem with the try-except construct on Dataflow - even if it's included, my Dataflow runner fails with the same ImportError. Do you have any idea why can this happen?

from tfx.

ruoyu90 avatar ruoyu90 commented on September 21, 2024

@mwalenia thanks for the notice! We are close to release 0.13 which will have Python 3.5 support.

from tfx.

mwalenia avatar mwalenia commented on September 21, 2024

@ruoyu90 Hi, when can I expect a fix for this issue? #91 seems to be stagnant, should I submit my own PR?

from tfx.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.