snowflake-labs / sfguide-data-engineering-with-snowpark-python Goto Github PK

In the Configure Credentials step, it is suggested that you specify the connection with a default role, warehouse and database. The creation of these objects, however, is not until the next step.

Inside utils/snowpark_utils.py

THIS WILL NOT WORK AS THE KEYS BETWEEN SNOWCLI AND SNOWPARK ARE DIFFERENT!!

    # Convert snowsql connection variable names to snowcli ones
    # session_config_dict = {
    #     snowsql_to_snowpark_config_mapping[k]: v.strip('"')
    #     for k, v in session_config.items()
    # }

WE NEED TO IETRATE OVER BOTH DICTIONARIES AT THE SAME TIME

    session_config_dict = {}
    for (k1,v1), (k2,v2) in zip(snowsql_to_snowpark_config_mapping.items(), session_config.items()):
        session_config_dict[k1] = v2.strip('"')

Also, inside steps/02_load_raw.py

perhaps make this more foolproof

  # current_dir = os.getcwd()
  current_dir = os.path.dirname(os.path.abspath(__file__))

When running the 02_load_raw.py file, there is an error as seen below.

`Traceback (most recent call last):
File ".\sfguide-data-engineering-with-snowpark-python\utils\snowpark_utils.py", line 74, in get_snowsql_config
session_config = config[connection_path]
File ".\miniconda3\envs\pysnowpark\lib\configparser.py", line 960, in getitem
raise KeyError(key)
KeyError: 'connections.dev'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "02_load_raw.py", line 76, in
session = snowpark_utils.get_snowpark_session()
File ".\sfguide-data-engineering-with-snowpark-python\utils\snowpark_utils.py", line 28, in get_snowpark_session
snowpark_config = get_snowsql_config()
File ".\sfguide-data-engineering-with-snowpark-python\utils\snowpark_utils.py", line 82, in get_snowsql_config
raise Exception(
Exception: Error getting snowsql config details`

This error occurs because the code looks for the variable connections.dev in my snowsql config file. In my case, this connection was not named dev but a more specific name.

[connections.snowpark_login] accountname = myaccount.west-europe.azure username = my_user password = my_password
Thus it cannot find login details in the config file. I fixed it for now by hardcoding my own naming at line 54 in the snowpark_utils.py file but there should either be a dynamic way to search for the login credentials or the quickstart should specify explicitly that you need to name the connection in this way.

Getting error while running following command:-

snow procedure create
Resolving any requirements from requirements.txt...
Comparing provided packages from Snowflake Anaconda...
No packages to manually resolve
Writing requirements.snowflake.txt file...
.
.
.
ProgrammingError: 001003 (42000): SQL compilation error:
syntax error line 1 at position 26 unexpected '~'.
parse error line 1 at position 37 near '76'.
parse error line 1 at position 43 near '84'.
parse error line 1 at position 48 near '116'.
syntax error line 1 at position 59 unexpected '/'.
syntax error line 1 at position 63 unexpected '.'.
syntax error line 1 at position 68 unexpected '@HOL_DB.HARMONIZED.deployments/orders_update_sp'.

Error at Snowflake looks like this :-

PUT file://C:\Users\PRANIL~1\AppData\Local\Temp\tmpn63b0aay/app.zip @HOL_DB.HARMONIZED.☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺_update_sp auto_compress=false overwrite=true

SQL compilation error: syntax error line 1 at position 26 unexpected '~'. parse error line 1 at position 37 near '76'. parse error line 1 at position 43 near '84'. parse error line 1 at position 48 near '116'. syntax error line 1 at position 59 unexpected '/'. syntax error line 1 at position 63 unexpected '.'. syntax error line 1 at position 68 unexpected '@HOL_DB.HARMONIZED.deployments/orders_update_sp'.

To align with best practices and to assure that session are really closed in case of errors examples should use snowpakr sesson in the following way:

    with Session.builder.getOrCreate() as session:
        print(main(session, *sys.argv[1:]))  # type: ignore

When running the 02_load_raw.py file, I encountered the following error:

Loading country Failed to execute query [queryID: 01accf15-0302-122a-0001-a636000250a6] use schema RAW_POS 002043 (02000): SQL compilation error: Object does not exist, or operation cannot be performed. Traceback (most recent call last): File "02_load_raw.py", line 78, in <module> load_all_raw_tables(session) File "02_load_raw.py", line 54, in load_all_raw_tables load_raw_table(session, tname=tname, s3dir=s3dir, schema=schema) File "02_load_raw.py", line 26, in load_raw_table session.use_schema(schema) File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\snowpark\session.py", line 1763, in use_schema self._use_object(schema, "schema") File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\snowpark\session.py", line 1799, in _use_object self._run_query(f"use {object_type} {object_name}") File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\snowpark\session.py", line 1231, in _run_query return self._conn.run_query( File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\snowpark\_internal\server_connection.py", line 102, in wrap raise ex line 346, in run_query results_cursor = self._cursor.execute(query, params=params, **kwargs) File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\connector\cursor.py", line 904, in execute Error.errorhandler_wrapper(self.connection, self, error_class, errvalue) File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\connector\errors.py", line 290, in errorhandler_wrapper handed_over = Error.hand_to_other_handler( File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\connector\errors.py", line 345, in hand_to_other_handler cursor.errorhandler(connection, cursor, error_class, error_value) File ".\miniconda3\envs\pysnowpark\lib\site-packages\snowflake\connector\errors.py", line 221, in default_errorhandler raise error_class( snowflake.connector.errors.ProgrammingError: 002043 (02000): SQL compilation error: Object does not exist, or operation cannot be performed.

After testing it out in my own Snowflake instance, I found out that the query lacks a specification of which database to search for this schema. I fixed the issue by replacing line 45 in the load_all_raw_tables function in the 02_load_raw.py file with this:

schema = 'HOL_DB.' + data['schema']

After this, the whole script executed successfully.