taosdata / taos-connector-jdbc Goto Github PK
View Code? Open in Web Editor NEWConnecting Java Application With a TDengine Database.
License: MIT License
Connecting Java Application With a TDengine Database.
License: MIT License
RestfulPreparedStatement 类中 line 23
RestfulPreparedStatement 解决的 this.rawSql = Utils.preprocessSql(sql);
被注释掉,导致spark 调用出现 查询出现bug 无法获取到数据
从 3.1.0 升级 到 3.2.0
Caused by: java.lang.NumberFormatException: For input string: "${project"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[na:na]
at java.base/java.lang.Integer.parseInt(Integer.java:654) ~[na:na]
at java.base/java.lang.Integer.parseInt(Integer.java:786) ~[na:na]
at com.taosdata.jdbc.AbstractDatabaseMetaData.<clinit>(AbstractDatabaseMetaData.java:30) ~[taos-jdbcdriver-3.2.0.jar:na]
我想知道这类 break change 从哪里可以看到?是需要每个开发人员体验踩坑的乐趣吗?
When we declare table with unsigned tinyint, and store value larger than max value of corresponding SIGNED type, negative number is return.
CREATE TABLE t1 (ts TIMESTAMP, f1 TINYINT UNSINGED, f2 SMALLINT UNSIGNED, f3 INT UNSIGNED, f4 BIGINT UNSIGNED);
INSERT INTO t1 (ts, f1, f2, f3, f4) VALUES (NOW, 255, 65535, 4294967295, POW(2, 64) - 1);
Just like the mysql-connector-j way, map TINY_UNSIGNED to Integer, SMALLINT_UNSIGNED to Integer, INT_UNSIGNED to Long, BIGINT_UNSIGNED to BigInteger.
UTINYINT
should use Short
instead of Byte
.USMALLINT
should use Integer
instead of Short
.UINT
should use Long
instead of Integer
.UBIGINT
should use BigInteger
instead of Long
private boolean isTableExist(Connection connection, String tableName) throws SQLException {
var dbname = connection.getCatalog();
var metadata = connection.getMetaData().unwrap(TSDBDatabaseMetaData.class);
try (var ignore = metadata.getColumns(dbname, "", tableName, "")) {
return true;
} catch (SQLException e) {
return false;
}
}
When running with the above code, exception is raised: TDengine ERROR (0x80002662): Fail to get table info, error: Invalid system table name
.
This issue was introduced with the latest jdbc connector, and the same piece of codes run ok with 3.2.4
version.
Would you please help to investigate?
WSClient核心线程数为什么是1,单线程池的作用是什么,调用很容易出错
TSDBJNIConnector类
private static void handleTimeZone(String posixTimeZoneStr) {
if (posixTimeZoneStr.startsWith("UTC") || posixTimeZoneStr.startsWith("GMT")) {
if (posixTimeZoneStr.length() == 3) {
TimeZone.setDefault(TimeZone.getTimeZone(ZoneId.of("GMT")));
return;
}
String regex = "^(UTC|GMT)([+-])(\\d+)$";
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(posixTimeZoneStr);
if (matcher.matches()) {
String op = matcher.group(2);
String hourStr = matcher.group(3);
// 这里+ 为什么要变成-
if (op.equals("+")) {
op = "-";
} else {
op = "+";
}
int hour = Integer.parseInt(hourStr);
if (hour > 18) {
return;
}
String timezone = String.format("GMT%s%02d:00", op, hour);
// 这里为什么要设置默认时区,导致我jvm设置user.timezone都给覆盖了
TimeZone.setDefault(TimeZone.getTimeZone(ZoneId.of(timezone)));
return;
}
}
if (posixTimeZoneStr.contains("/")) {
TimeZone.setDefault(TimeZone.getTimeZone(ZoneId.of(posixTimeZoneStr)));
}
}
does it already support geometry feature?
发行版Releases中能不能添加打包好的jar?总共3个,今天下载依赖并打包浪费了半天时间。
taos-jdbcdriver-3.1.0.jar
taos-jdbcdriver-3.1.0-dist.jar
taos-jdbcdriver-3.1.0-sources.jar
PoolableConnection conn = getConnection(serverConfig);
DatabaseMetaData md = conn.getMetaData();
// TABLE_CAT:`; TABLE_SCHEM:`; TABLE_NAME:`
String db = serverConfig.getDatabase();
ResultSet rs = md.getTables(db, null, "%", new String[] {"TABLE","VIEW"});
md.getTables("db", null, "%", new String[] {"TABLE","VIEW"});
- returns the tables "db" only, with the TABLE_CAT populated
md.getTables(null, null, "%", new String[] {"TABLE","VIEW"});
- returns all tables but does NOT populate TABLE_CAT
Is there a way to fetch all tables in all schemas AND populate the TABLE_CAT column?
Closely related test:
Also your information_schema tables are prefixed INS_XXXXXX, is there a reason for that?
https://docs.tdengine.com/taos-sql/meta/#ins_columns
环境
CentOS 7
操作系统时区 东8区
jdk 1.8.0_201-b09
spring boot 1.5.18
taos 客户端 3.2.2.0
jdbc 版本 3.2.7
连接池均使用 ali druid (com.alibaba:druid:1.1.9/ com.alibaba:druid:1.0.26)
阿里云 RDS mysql 5.6 数据库默认时区东8区
jdbc驱动: mysql-connector-java:5.1.47
ORM spring data JPA
连接池配置
spring.datasource.druid.admin.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.druid.admin.url=jdbc:mysql://host/db?useUnicode=true&characterEncoding=utf-8&supportBigNumbers=true
taos 连接池配置
没有使用ORM, 直接操作
spring.datasource.druid.taos.driver-class-name=com.taosdata.jdbc.TSDBDriver
spring.datasource.druid.taos.url=jdbc:TAOS://:/?locale=UTF-8&charset=UTF-8&timezone=UTC
taos数据库的写入采用拼接sql字符串的方式,spring JDBCTemplate#update(sql),没有使用ORM
现象:
MySQL数据库的某个数据表字段类型 DATETIME
Java 实体对应字段类型(2个类型都测试过)java.util.Date 或者 org.joda.time.LocalDateTime
存入MySQL数据库时,时间会晚8小时,
如果 把taos jdbc 3.2.7驱动 的数据库连接串中的 timezone=UTC 去掉, 则MySQL数据库 DATETIME 写入正常
如果用taos jdbc 3.1.0驱动, timezone=UTC可以存在且不影响MySQL的写入。
该Java实体:从未向TDengine写入或读取, 即不对应TDengine中的表,只对应MySQL的表
另一个环境是 spring boot 2.4.0, jdk 8, jdbc驱动:mysql-connector-java:8.0.22
driver-class-name=com.mysql.cj.jdbc.Driver
url= jdbc:mysql://host/db?serverTimezone=CTT&useUnicode=true&characterEncoding=utf-8&supportBigNumbers=true
taos jdbc url 中也有 timezone=UTC
mysql 连接串中有 serverTimezone=CTT结果时间早8小时。
如果taosjdbc驱动换成3.1.0则时间正常
直接使用字符串 拼接的方式提交SQL,是否会有SQL注入风险呢?
// DLL_EXPORT const char *tmq_get_table_name(TAOS_RES *res);
private native String tmqGetTableName(long res);
在订阅超级表和数据库时,消费到底数据中没有数据所属的表名,经查,上面这个方法返回空。同样的订阅,在c++和c#中可以取到表名
使用JNI模式,无法排除fastjson依赖
<dependency> <groupId>com.taosdata.jdbc</groupId> <artifactId>taos-jdbcdriver</artifactId> <version>3.2.4</version> <exclusions> <exclusion> <artifactId>fastjson</artifactId> <groupId>com.alibaba</groupId> </exclusion> </exclusions> </dependency>
原因在于 TSDBJNIConnector 中引用了fastjson
JSONObject configJSON = new JSONObject(); for (String key : props.stringPropertyNames()) { configJSON.put(key, props.getProperty(key)); } setConfigImp(configJSON.toJSONString());
JNI中就这么一小段试用了fastjson,完全没有必要
目前com.taosdata.jdbc.TSDBJNIConnector
类强制使用System.loadLibrary
方法类加载jni动态库,代码如下:
/**
* JNI connector
*/
public class TSDBJNIConnector {
private static final Object LOCK = new Object();
private static volatile boolean isInitialized;
private final TaosInfo taosInfo = TaosInfo.getInstance();
protected long taos = TSDBConstants.JNI_NULL_POINTER; // Connection pointer used in C
private boolean isResultsetClosed; // result set status in current connection
private int affectedRows = -1;
static {
System.loadLibrary("taos");
}
...
无法将jni动态库文件放入jar包内或其他配置目录来自由加载动态库,部署时必须要将动态库放到java.library.path
内 或 将jni动态库文件所在目录指定到java.library.path
,封装不够彻底,运维较麻烦
java.library.path
内的动态库只在jvm启动时加载,项目启动后无法加载(jdk11前可以用反射方式强制jdk重新加载至少在jdk17后不行)。
在执行System.loadLibrary
做一下判断,如果识别动态库已经载入jvm就不执行System.loadLibrary
方法,封装者就能在使用TSDBJNIConnector
前自己使用System.load
方法加载动态库文件
windows系统 在mybatis环境中使用jdbc3.0.2驱动,使用查询sql方法如下, select last(ts) as real_time , last(v) as value_collect from test 映射到对象上的realTime和valueCollect。 预期realTime和valueCollect正常映射到数据。但是结果确是null,增加ts和v参数, 发现数据还是映射在ts和v参数上。 该问题在linux系统中是正常的。于是调试发现
unMappedColumnNames在linux和windows中运行的结果并不一样。 linux是正确的,所以未出问题。而windows并没有用别名映射对象。而仍然使用的ts和v进行对象赋值。
如果数据库的时间戳精度设置为微秒或者纳秒,在java中通过tmq获取订阅的数据时,时间戳不能被正确转换为正确的时间格式,都会按照毫秒的精度转换,所以时间段年份会变的特别大。经过跟踪发现,在com.taosdata.jdb.TSDBJNIConnector 中的
/**
* Get Result Time Precision.
*
* @return 0: ms, 1: us, 2: ns
*/
public int getResultTimePrecision(long sqlObj) {
return this.getResultTimePrecisionImp(this.taos, sqlObj);
}
private native int getResultTimePrecisionImp(long connection, long result);
这个方法获得的精度总是0,也就是毫秒。
Hello!
As a continuation of this ticket dbeaver/dbeaver#22161
I can create tables with upper-case symbols using quotes.
But I can't get columns/primary key info from the driver because of the error:
!ENTRY org.jkiss.dbeaver.model 4 0 2024-01-26 08:18:12.906
!MESSAGE TDengine ERROR (0x2603): sql: describe power.device_property_27NbfMsBLLa_d0000000001_e_1, desc: Table does not exist
!STACK 0
java.sql.SQLException: TDengine ERROR (0x2603): sql: describe power.device_property_27NbfMsBLLa_d0000000001_e_1, desc: Table does not exist
at com.taosdata.jdbc.TSDBError.createSQLException(TSDBError.java:95)
at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:97)
at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:78)
at com.taosdata.jdbc.rs.RestfulStatement.executeQuery(RestfulStatement.java:39)
at com.taosdata.jdbc.AbstractDatabaseMetaData.getColumns(AbstractDatabaseMetaData.java:1098)
at com.taosdata.jdbc.rs.RestfulDatabaseMetaData.getColumns(RestfulDatabaseMetaData.java:78)
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCDatabaseMetaDataImpl.getColumns(JDBCDatabaseMetaDataImpl.java:1152)
at org.jkiss.dbeaver.ext.generic.model.meta.GenericMetaModel.prepareTableColumnLoadStatement(GenericMetaModel.java:708)
at org.jkiss.dbeaver.ext.generic.model.TableCache.prepareChildrenStatement(TableCache.java:81)
Because you do not add quotes for tables in quotes:
Different drivers read metadata in different ways, and I see that MySQL driver (as an example) adds quotes for object names:
taos service linux x86 3.0.1.7
taos windows cline3.0.1.7
jdbc-driver:3.0.2
jdk8 异常:
Exception in thread "main" java.lang.NoSuchMethodError: setByteArray
at com.taosdata.jdbc.TSDBJNIConnector.initImp(Native Method)
at com.taosdata.jdbc.TSDBJNIConnector.init(TSDBJNIConnector.java:43)
at com.taosdata.jdbc.TSDBDriver.connect(TSDBDriver.java:157)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.taosdata.example.JdbcDemo.init(JdbcDemo.java:44)
at com.taosdata.example.JdbcDemo.main(JdbcDemo.java:24)
jdk17异常:
Exception in thread "main" java.lang.NoSuchMethodError: Lcom/taosdata/jdbc/TSDBResultSetBlockData;.setByteArray(II[B)V
at com.taosdata.jdbc.TSDBJNIConnector.initImp(Native Method)
at com.taosdata.jdbc.TSDBJNIConnector.init(TSDBJNIConnector.java:43)
at com.taosdata.jdbc.TSDBDriver.connect(TSDBDriver.java:157)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:681)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
at com.taosdata.example.JdbcDemo.init(JdbcDemo.java:44)
at com.taosdata.example.JdbcDemo.main(JdbcDemo.java:24)
initImp为native方法TSDBResultSetBlockData.setByteArray也存在,不知什么原因
TDEngine版本:3.0.3.1
taos-jdbcdriver版本:3.1.0
超级表建表语句:
CREATE STABLE stable (ts TIMESTAMP, c1 NCHAR(20), c2 NCHAR(100), c3 NCHAR(20), c4 BIGINT) TAGS (t1 NCHAR(40));
参数绑定写入数据代码片段:
String sql = "INSERT INTO ? USING stable TAGS (?) (ts,c1,c2,c3) VALUES(?,?,?,?)";
try (TSDBPreparedStatement pstmt = connection.prepareStatement(sql).unwrap(TSDBPreparedStatement.class)) {
for (Map.Entry<String, List<DTO>> entry : map.entrySet()) {
String t1 = entry.getKey();
List<DTO> dtoList = entry.getValue();
pstmt.setTableName("p_" + t1);
pstmt.setTagNString(0, t1);
int size = dtoList.size();
ArrayList<Long> tsList = new ArrayList<>(size);
ArrayList<String> c1List = new ArrayList<>(size);
ArrayList<String> c2List = new ArrayList<>(size);
ArrayList<String> c3List = new ArrayList<>(size);
for (DTO dto : dtoList) {
tsList.add(dto.getTs());
c1List.add(dto.getC1());
c2List.add(dto.getC2());
c3List.add(dto.getC3());
}
pstmt.setTimestamp(0, tsList);
pstmt.setNString(1, c1List, 20);
pstmt.setNString(2, c2List, 100);
pstmt.setNString(3, c3List, 20);
pstmt.columnDataAddBatch();
}
pstmt.columnDataExecuteBatch();
}
报错信息:
java.sql.SQLException: TDengine ERROR (2350): failed to execute batch bind, reason: Table already exists
at com.taosdata.jdbc.TSDBError.createSQLException(TSDBError.java:86)
at com.taosdata.jdbc.TSDBJNIConnector.executeBatch(TSDBJNIConnector.java:300)
at com.taosdata.jdbc.TSDBPreparedStatement.columnDataExecuteBatch(TSDBPreparedStatement.java:899)
项目部署在Tomcat上,使用Druid连接池,配taos.jdbc,JDK1.8,TD2.6,项目运行一段时间后,JVM会直接崩溃,有没有人有相同的情况?如何解决的?
--------------- T H R E A D ---------------
Current thread is native thread
siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007f0ea4000450
............
Register to memory mapping:
RAX=0x00007f0ea4000450 is an unknown value
RBX=0x0000000000000000 is an unknown value
RCX=0x00007f0ea412e7a8 is an unknown value
RDX=0x00007f0e47bf7d00 is an unknown value
RSP=0x00007f0e47bf7c88 is an unknown value
RBP=0x00007f0e47bf7cb0 is an unknown value
RSI=0x00007f0ea412e7a8 is an unknown value
RDI=0x00007f0e47bf7d00 is an unknown value
R8 =0x0000000000000008 is an unknown value
R9 =0x0003e9e0eced3bfa is an unknown value
R10=0x000000000000001b is an unknown value
R11=0x00007f0f85b91f10: <offset 0x169f10> in /lib64/libc.so.6 at 0x00007f0f85a28000
R12=0x00007f0e51b8ca6e is pointing into the stack for thread: 0x00007f0eb8460000
R13=0x00007f0e51b8ca6f is pointing into the stack for thread: 0x00007f0eb8460000
R14=0x00007f0e9025fe80 is an unknown value
R15=0x00007f0e51b8cb50 is pointing into the stack for thread: 0x00007f0eb8460000
Stack: [0x00007f0e473f9000,0x00007f0e47bf9000], sp=0x00007f0e47bf7c88, free space=8187k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C 0x00007f0ea4000450
C [libtaos.so.2.6.0.34+0x1b6330] rpcProcessIncomingMsg+0x49e
C [libtaos.so.2.6.0.34+0x1b5d3f] rpcProcessMsgFromPeer+0x475
C [libtaos.so.2.6.0.34+0x1bae74] taosProcessTcpData+0x3d3
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.