Code Monkey home page Code Monkey logo

taos-connector-jdbc's People

Contributors

chaixuqing avatar dapan1121 avatar dependabot[bot] avatar dingbo8128 avatar duankuanjun avatar evpeople avatar freemine avatar gccgdb1234 avatar guanshengliang avatar hjxilinx avatar huolibo avatar hzcheng avatar jiacy-jcy avatar lonwolonwo avatar plum-lihui avatar sangshuduo avatar shenglian-zhou avatar sheyanjie-qq avatar sheyanjie-taos avatar stephenkgu avatar sunguofeng avatar sunpe avatar wangmm0220 avatar weicz1221 avatar xiaopingcs avatar xieyinglin avatar yihaodeng avatar yijiue avatar zhaoyanggh avatar zyyang90 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

taos-connector-jdbc's Issues

3.2.0版本启动就报错

从 3.1.0 升级 到 3.2.0

Caused by: java.lang.NumberFormatException: For input string: "${project"
        at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[na:na]
        at java.base/java.lang.Integer.parseInt(Integer.java:654) ~[na:na]
        at java.base/java.lang.Integer.parseInt(Integer.java:786) ~[na:na]
        at com.taosdata.jdbc.AbstractDatabaseMetaData.<clinit>(AbstractDatabaseMetaData.java:30) ~[taos-jdbcdriver-3.2.0.jar:na]

我想知道这类 break change 从哪里可以看到?是需要每个开发人员体验踩坑的乐趣吗?

Bug: unsigned tinyint/smallint/int/bigint issue

Senario:

When we declare table with unsigned tinyint, and store value larger than max value of corresponding SIGNED type, negative number is return.

Reproduce

CREATE TABLE t1 (ts TIMESTAMP, f1 TINYINT UNSINGED, f2 SMALLINT UNSIGNED, f3 INT UNSIGNED, f4 BIGINT UNSIGNED);
INSERT INTO t1 (ts, f1, f2, f3, f4) VALUES (NOW, 255, 65535, 4294967295, POW(2, 64) - 1);

Solution:

Just like the mysql-connector-j way, map TINY_UNSIGNED to Integer, SMALLINT_UNSIGNED to Integer, INT_UNSIGNED to Long, BIGINT_UNSIGNED to BigInteger.

  1. UTINYINT should use Short instead of Byte.
  2. USMALLINT should use Integer instead of Short.
  3. UINT should use Long instead of Integer.
  4. UBIGINT should use BigInteger instead of Long

Bug: TSDBDatabaseMetaData get metadata return "invalid system table name"

Environment:

  • server version: tdengine-3.1.0.3
  • jdbc version: 3.2.8
  • jdk version: JDK 17

Reproduce:

  private boolean isTableExist(Connection connection, String tableName) throws SQLException {
    var dbname = connection.getCatalog();
    var metadata = connection.getMetaData().unwrap(TSDBDatabaseMetaData.class);
    try (var ignore = metadata.getColumns(dbname, "", tableName, "")) {
      return true;
    } catch (SQLException e) {
      return false;
    }
  }

When running with the above code, exception is raised: TDengine ERROR (0x80002662): Fail to get table info, error: Invalid system table name.

This issue was introduced with the latest jdbc connector, and the same piece of codes run ok with 3.2.4 version.

Would you please help to investigate?

WSClient核心线程数

WSClient核心线程数为什么是1,单线程池的作用是什么,调用很容易出错

时区错误问题

TSDBJNIConnector类

private static void handleTimeZone(String posixTimeZoneStr) {
        if (posixTimeZoneStr.startsWith("UTC") || posixTimeZoneStr.startsWith("GMT")) {
            if (posixTimeZoneStr.length() == 3) {
                TimeZone.setDefault(TimeZone.getTimeZone(ZoneId.of("GMT")));
                return;
            }

            String regex = "^(UTC|GMT)([+-])(\\d+)$";
            Pattern pattern = Pattern.compile(regex);
            Matcher matcher = pattern.matcher(posixTimeZoneStr);
            if (matcher.matches()) {
                String op = matcher.group(2);
                String hourStr = matcher.group(3);
                // 这里+ 为什么要变成-
                if (op.equals("+")) {
                    op = "-";
                } else {
                    op = "+";
                }

                int hour = Integer.parseInt(hourStr);
                if (hour > 18) {
                    return;
                }

                String timezone = String.format("GMT%s%02d:00", op, hour);
                // 这里为什么要设置默认时区,导致我jvm设置user.timezone都给覆盖了
                TimeZone.setDefault(TimeZone.getTimeZone(ZoneId.of(timezone)));
                return;
            }
        }

        if (posixTimeZoneStr.contains("/")) {
            TimeZone.setDefault(TimeZone.getTimeZone(ZoneId.of(posixTimeZoneStr)));
        }

    }

JDBC Meta data


			PoolableConnection conn = getConnection(serverConfig);
			DatabaseMetaData md = conn.getMetaData();
			//  TABLE_CAT:`; TABLE_SCHEM:`; TABLE_NAME:`
			String db = serverConfig.getDatabase();
			ResultSet rs = md.getTables(db, null, "%", new String[] {"TABLE","VIEW"});

md.getTables("db", null, "%", new String[] {"TABLE","VIEW"}); - returns the tables "db" only, with the TABLE_CAT populated
md.getTables(null, null, "%", new String[] {"TABLE","VIEW"}); - returns all tables but does NOT populate TABLE_CAT

Is there a way to fetch all tables in all schemas AND populate the TABLE_CAT column?
Closely related test:

ResultSet rs = metaData.getTables(dbName, "", null, null);
ResultSetMetaData meta = rs.getMetaData();
Assert.assertNotNull(rs);
rs.next();

Also your information_schema tables are prefixed INS_XXXXXX, is there a reason for that?
https://docs.tdengine.com/taos-sql/meta/#ins_columns

3.2.7版本的连接串中的时区设置影响mysql的连接时区设置

环境
CentOS 7
操作系统时区 东8区
jdk 1.8.0_201-b09

spring boot 1.5.18

taos 客户端 3.2.2.0
jdbc 版本 3.2.7

连接池均使用 ali druid (com.alibaba:druid:1.1.9/ com.alibaba:druid:1.0.26)

阿里云 RDS mysql 5.6 数据库默认时区东8区
jdbc驱动: mysql-connector-java:5.1.47
ORM spring data JPA
连接池配置
spring.datasource.druid.admin.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.druid.admin.url=jdbc:mysql://host/db?useUnicode=true&characterEncoding=utf-8&supportBigNumbers=true

taos 连接池配置
没有使用ORM, 直接操作
spring.datasource.druid.taos.driver-class-name=com.taosdata.jdbc.TSDBDriver
spring.datasource.druid.taos.url=jdbc:TAOS://:/?locale=UTF-8&charset=UTF-8&timezone=UTC

taos数据库的写入采用拼接sql字符串的方式,spring JDBCTemplate#update(sql),没有使用ORM

现象:
MySQL数据库的某个数据表字段类型 DATETIME

Java 实体对应字段类型(2个类型都测试过)java.util.Date 或者 org.joda.time.LocalDateTime
存入MySQL数据库时,时间会晚8小时,

如果 把taos jdbc 3.2.7驱动 的数据库连接串中的 timezone=UTC 去掉, 则MySQL数据库 DATETIME 写入正常

如果用taos jdbc 3.1.0驱动, timezone=UTC可以存在且不影响MySQL的写入。

该Java实体:从未向TDengine写入或读取, 即不对应TDengine中的表,只对应MySQL的表

另一个环境是 spring boot 2.4.0, jdk 8, jdbc驱动:mysql-connector-java:8.0.22
driver-class-name=com.mysql.cj.jdbc.Driver
url= jdbc:mysql://host/db?serverTimezone=CTT&useUnicode=true&characterEncoding=utf-8&supportBigNumbers=true
taos jdbc url 中也有 timezone=UTC
mysql 连接串中有 serverTimezone=CTT结果时间早8小时。
如果taosjdbc驱动换成3.1.0则时间正常

3.0版本与ibatis使用int=0和boolean=false对象赋值为null

版本3.0 当和mybatis plus一起使用时, 该日志注解会导致 int是0和boolean是false类型的参数都被赋值为null,去掉日志打印注解则不会出现。
mybatis-plus:
configuration:
log-impl: org.apache.ibatis.logging.stdout.StdOutImpl
调试时发现,加上日志注解会导致TSDBResultSetBlockData get方法提前赋值wasNull = true。 导致ibatis对象赋值时判断出现问题
image

windows上客户端和jdbc驱动在windows环境上ts时间查询问题

  1. 在linxu客户端上插入一条数据并查询,2022/11/18 9:00:00
    image
  2. 在windows客户端上执行查询, 时间少了16个小时, 变成了2022/11/17 17:00:00
    image
  3. 使用jdbc驱动在windows上查询,使用字符串查询必须写少16个小时的时间, 但使用时间戳查询,是正确时间的时间戳,最终都能查出数据。 但得到的反馈结果的ts都是正确的。同样的配置在linux上执行查询,字符串和时间戳使用正确的时间查询都没有问题。时区确定没有问题 jdbc配置 ?charset=UTF-8&locale=en_US.UTF-8&timezone=UTC-8
    image

TMQConnector不能取到表名

// DLL_EXPORT const char *tmq_get_table_name(TAOS_RES *res);
private native String tmqGetTableName(long res);
在订阅超级表和数据库时,消费到底数据中没有数据所属的表名,经查,上面这个方法返回空。同样的订阅,在c++和c#中可以取到表名

移除fastjson后JNI启动报错

使用JNI模式,无法排除fastjson依赖
<dependency> <groupId>com.taosdata.jdbc</groupId> <artifactId>taos-jdbcdriver</artifactId> <version>3.2.4</version> <exclusions> <exclusion> <artifactId>fastjson</artifactId> <groupId>com.alibaba</groupId> </exclusion> </exclusions> </dependency>
原因在于 TSDBJNIConnector 中引用了fastjson
JSONObject configJSON = new JSONObject(); for (String key : props.stringPropertyNames()) { configJSON.put(key, props.getProperty(key)); } setConfigImp(configJSON.toJSONString());

JNI中就这么一小段试用了fastjson,完全没有必要

[功能建议] jni动态库支持调用方用System.load()方法自己加载

现状

目前com.taosdata.jdbc.TSDBJNIConnector 类强制使用System.loadLibrary方法类加载jni动态库,代码如下:

/**
 * JNI connector
 */
public class TSDBJNIConnector {
    private static final Object LOCK = new Object();
    private static volatile boolean isInitialized;

    private final TaosInfo taosInfo = TaosInfo.getInstance();
    protected long taos = TSDBConstants.JNI_NULL_POINTER;     // Connection pointer used in C
    private boolean isResultsetClosed;      // result set status in current connection
    private int affectedRows = -1;

    static {
        System.loadLibrary("taos");
    }
...

问题

无法将jni动态库文件放入jar包内或其他配置目录来自由加载动态库,部署时必须要将动态库放到java.library.path内 或 将jni动态库文件所在目录指定到java.library.path,封装不够彻底,运维较麻烦

原因

java.library.path内的动态库只在jvm启动时加载,项目启动后无法加载(jdk11前可以用反射方式强制jdk重新加载至少在jdk17后不行)。

建议

在执行System.loadLibrary做一下判断,如果识别动态库已经载入jvm就不执行System.loadLibrary方法,封装者就能在使用TSDBJNIConnector前自己使用System.load方法加载动态库文件

版本3.0.0及以上 windows环境中使用last 函数配合别名在mybatis中无法映射

windows系统 在mybatis环境中使用jdbc3.0.2驱动,使用查询sql方法如下, select last(ts) as real_time , last(v) as value_collect from test 映射到对象上的realTime和valueCollect。 预期realTime和valueCollect正常映射到数据。但是结果确是null,增加ts和v参数, 发现数据还是映射在ts和v参数上。 该问题在linux系统中是正常的。于是调试发现
image
unMappedColumnNames在linux和windows中运行的结果并不一样。 linux是正确的,所以未出问题。而windows并没有用别名映射对象。而仍然使用的ts和v进行对象赋值。

tmq订阅数据时时间戳精度问题

如果数据库的时间戳精度设置为微秒或者纳秒,在java中通过tmq获取订阅的数据时,时间戳不能被正确转换为正确的时间格式,都会按照毫秒的精度转换,所以时间段年份会变的特别大。经过跟踪发现,在com.taosdata.jdb.TSDBJNIConnector 中的
/**
* Get Result Time Precision.
*
* @return 0: ms, 1: us, 2: ns
*/
public int getResultTimePrecision(long sqlObj) {
return this.getResultTimePrecisionImp(this.taos, sqlObj);
}

private native int getResultTimePrecisionImp(long connection, long result);

这个方法获得的精度总是0,也就是毫秒。

Add quotes for table names in the AbstractDatabaseMetaData for getColumns and getPrimaryKeys

Hello!

As a continuation of this ticket dbeaver/dbeaver#22161

I can create tables with upper-case symbols using quotes.
But I can't get columns/primary key info from the driver because of the error:

!ENTRY org.jkiss.dbeaver.model 4 0 2024-01-26 08:18:12.906
!MESSAGE TDengine ERROR (0x2603): sql: describe power.device_property_27NbfMsBLLa_d0000000001_e_1, desc: Table does not exist
!STACK 0
java.sql.SQLException: TDengine ERROR (0x2603): sql: describe power.device_property_27NbfMsBLLa_d0000000001_e_1, desc: Table does not exist
	at com.taosdata.jdbc.TSDBError.createSQLException(TSDBError.java:95)
	at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:97)
	at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:78)
	at com.taosdata.jdbc.rs.RestfulStatement.executeQuery(RestfulStatement.java:39)
	at com.taosdata.jdbc.AbstractDatabaseMetaData.getColumns(AbstractDatabaseMetaData.java:1098)
	at com.taosdata.jdbc.rs.RestfulDatabaseMetaData.getColumns(RestfulDatabaseMetaData.java:78)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCDatabaseMetaDataImpl.getColumns(JDBCDatabaseMetaDataImpl.java:1152)
	at org.jkiss.dbeaver.ext.generic.model.meta.GenericMetaModel.prepareTableColumnLoadStatement(GenericMetaModel.java:708)
	at org.jkiss.dbeaver.ext.generic.model.TableCache.prepareChildrenStatement(TableCache.java:81)

Because you do not add quotes for tables in quotes:

ResultSet rs = stmt.executeQuery("describe " + catalog + "." + table)) {

ResultSet rs = stmt.executeQuery("describe " + db + "." + s)) {

Different drivers read metadata in different ways, and I see that MySQL driver (as an example) adds quotes for object names:

2024-01-26 10_29_26-DBeaver  C__Users_anvolka_work_DbeaverP_dbeaver-workspace-idea  - C__Users_anvol

运行官方JDBCDemo

  • taos service linux x86 3.0.1.7

  • taos windows cline3.0.1.7

  • jdbc-driver:3.0.2

  • jdk8 异常:
    Exception in thread "main" java.lang.NoSuchMethodError: setByteArray
    at com.taosdata.jdbc.TSDBJNIConnector.initImp(Native Method)
    at com.taosdata.jdbc.TSDBJNIConnector.init(TSDBJNIConnector.java:43)
    at com.taosdata.jdbc.TSDBDriver.connect(TSDBDriver.java:157)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:208)
    at com.taosdata.example.JdbcDemo.init(JdbcDemo.java:44)
    at com.taosdata.example.JdbcDemo.main(JdbcDemo.java:24)

  • jdk17异常:
    Exception in thread "main" java.lang.NoSuchMethodError: Lcom/taosdata/jdbc/TSDBResultSetBlockData;.setByteArray(II[B)V
    at com.taosdata.jdbc.TSDBJNIConnector.initImp(Native Method)
    at com.taosdata.jdbc.TSDBJNIConnector.init(TSDBJNIConnector.java:43)
    at com.taosdata.jdbc.TSDBDriver.connect(TSDBDriver.java:157)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:681)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
    at com.taosdata.example.JdbcDemo.init(JdbcDemo.java:44)
    at com.taosdata.example.JdbcDemo.main(JdbcDemo.java:24)
    initImp为native方法TSDBResultSetBlockData.setByteArray也存在,不知什么原因

参数绑定写入数据报错Table already exists

TDEngine版本:3.0.3.1
taos-jdbcdriver版本:3.1.0
超级表建表语句:
CREATE STABLE stable (ts TIMESTAMP, c1 NCHAR(20), c2 NCHAR(100), c3 NCHAR(20), c4 BIGINT) TAGS (t1 NCHAR(40));
参数绑定写入数据代码片段:

String sql = "INSERT INTO ? USING stable TAGS (?) (ts,c1,c2,c3) VALUES(?,?,?,?)";
try (TSDBPreparedStatement pstmt = connection.prepareStatement(sql).unwrap(TSDBPreparedStatement.class)) {
    for (Map.Entry<String, List<DTO>> entry : map.entrySet()) {
        String t1 = entry.getKey();
        List<DTO> dtoList = entry.getValue();
        pstmt.setTableName("p_" + t1);
        pstmt.setTagNString(0, t1);
        int size = dtoList.size();
        ArrayList<Long> tsList = new ArrayList<>(size);
        ArrayList<String> c1List = new ArrayList<>(size);
        ArrayList<String> c2List = new ArrayList<>(size);
        ArrayList<String> c3List = new ArrayList<>(size);
        for (DTO dto : dtoList) {
            tsList.add(dto.getTs());
            c1List.add(dto.getC1());
            c2List.add(dto.getC2());
            c3List.add(dto.getC3());
        }
        pstmt.setTimestamp(0, tsList);
        pstmt.setNString(1, c1List, 20);
        pstmt.setNString(2, c2List, 100);
        pstmt.setNString(3, c3List, 20);
        pstmt.columnDataAddBatch();
    }
    pstmt.columnDataExecuteBatch();
}

报错信息:

java.sql.SQLException: TDengine ERROR (2350): failed to execute batch bind, reason: Table already exists
	at com.taosdata.jdbc.TSDBError.createSQLException(TSDBError.java:86)
	at com.taosdata.jdbc.TSDBJNIConnector.executeBatch(TSDBJNIConnector.java:300)
	at com.taosdata.jdbc.TSDBPreparedStatement.columnDataExecuteBatch(TSDBPreparedStatement.java:899)

wasnull方法返回值异常

335321201476dd4d63df46d27972f26
!= null是不是要置为 false
896917f22dee7a7932984b6fdc3a189
使用springjdbctemplate
8d05250508bc20e3b927e14cf0f6d8b
发现wasnull返回true而实际有值 导致查询错误

JVM 崩溃

项目部署在Tomcat上,使用Druid连接池,配taos.jdbc,JDK1.8,TD2.6,项目运行一段时间后,JVM会直接崩溃,有没有人有相同的情况?如何解决的?

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f0ea4000450, pid=18570, tid=0x00007f0e47bf8700

JRE version: Java(TM) SE Runtime Environment (8.0_271-b09) (build 1.8.0_271-b09)

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.271-b09 mixed mode linux-amd64 compressed oops)

Problematic frame:

C 0x00007f0ea4000450

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please visit:

http://bugreport.java.com/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread is native thread

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007f0ea4000450

............

Register to memory mapping:

RAX=0x00007f0ea4000450 is an unknown value
RBX=0x0000000000000000 is an unknown value
RCX=0x00007f0ea412e7a8 is an unknown value
RDX=0x00007f0e47bf7d00 is an unknown value
RSP=0x00007f0e47bf7c88 is an unknown value
RBP=0x00007f0e47bf7cb0 is an unknown value
RSI=0x00007f0ea412e7a8 is an unknown value
RDI=0x00007f0e47bf7d00 is an unknown value
R8 =0x0000000000000008 is an unknown value
R9 =0x0003e9e0eced3bfa is an unknown value
R10=0x000000000000001b is an unknown value
R11=0x00007f0f85b91f10: <offset 0x169f10> in /lib64/libc.so.6 at 0x00007f0f85a28000
R12=0x00007f0e51b8ca6e is pointing into the stack for thread: 0x00007f0eb8460000
R13=0x00007f0e51b8ca6f is pointing into the stack for thread: 0x00007f0eb8460000
R14=0x00007f0e9025fe80 is an unknown value
R15=0x00007f0e51b8cb50 is pointing into the stack for thread: 0x00007f0eb8460000

Stack: [0x00007f0e473f9000,0x00007f0e47bf9000], sp=0x00007f0e47bf7c88, free space=8187k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C 0x00007f0ea4000450
C [libtaos.so.2.6.0.34+0x1b6330] rpcProcessIncomingMsg+0x49e
C [libtaos.so.2.6.0.34+0x1b5d3f] rpcProcessMsgFromPeer+0x475
C [libtaos.so.2.6.0.34+0x1bae74] taosProcessTcpData+0x3d3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.