Comments (1)
问题原因:
1.datax在运行时将异常数据打印到 log中导致log很大
2.netty在响应时候会读取log文件,导致Response entity too large
解决方法:
第一种:.构建datax.json 时候添加 errorLimit阈值,如果大于阈值任务自动失败
具体操作 json中添加
"errorLimit": {
"record": 10, //脏数据行数
"percentage": 0.05 // 脏数据百分比
}
第二种:将log日志格式修改,减小log体积
原来 2020-03-13 18:45:51 [com.wugui.datatx.core.thread.JobThread#run]-[130]-[Thread-18] <br>----------- datax-web job execute start -----------<br>----------- Param:
现在 2020-03-13 18:56:41 [JobThread.run-130] <br>----------- datax-web job execute start -----------<br>----------- Param:
具体操作:修改JobLogger中logDetail方法
第三种:增大netty response阈值(原来是5x1024x1024=5M),现在修改为20M
具体操作 修改NettyHttpConnectClient和NettyHttpServer中HttpObjectAggregator值
from datax-web.
Related Issues (20)
- 达梦数据库怎么连接
- 怎么去掉前段的登录校验啊
- 希望添加任务依赖功能 HOT 2
- 2.1.2-release版本 BuildCommand.buildDataXCustomParam未判空,导致空customParam 时,执行 指令不成功 HOT 2
- [Feature] 希望增加对TxtFileReader 和 TxtFileWriter的支持
- [QUESTION] I wonder konw wheather datax-web can deploy on window-server. HOT 1
- [性能] 当我单个或批量构建任务时,由于数据库schema中表的数量较大,导致浏览器直接卡死
- 2.1.2 release 任务执行时报错 datax.py [options] job-url-or-path 参数为空 HOT 1
- [QUESTION] 这项目还有人在维护么?好多 PR 都没合并。 HOT 1
- org.postgresql.util.PSQLException: Invalid or unsupported by client SCRAM mechanisms
- 希望能加上自定义数据源功能 HOT 2
- 创建任务时选择mongo数据库出现bug
- 希望页面增加一个刷新的功能
- 我的怎么没有时间增量呢 HOT 1
- 可以提供一个docker文件来安装吗,或者弄一个docker镜像拉取搭建 [Feature]
- java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens HOT 1
- [Feature]希望任务管理-编辑任务 能像任务构建一样编辑任务
- [BUG] Security Vulnerability - Action Required: XSS vulnerability in the newest version of the datax-web
- Docker镜像linshellfeng/datax_web 3.0.1 部署启动失败
- [QUESTION] datax-web executer error : Could not find or load main class com.alibaba.datax.core.Engine
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datax-web.