编辑
2022-05-31
ETL工具
00
请注意,本文编写于 685 天前,最后修改于 576 天前,其中某些信息可能已经过时。

目录

MySQL2MongoDB
注意事项

介绍公司实际项目实战,DATAX离线数据迁移。

MySQL2MongoDB

1.Josn配置文件可能加载失败,需要核对各参数,严格按照官方样例写参数,即使是Json格式也可能解析失败!

2.源库单表数据量过大时一定要加主键,加分片键 "splitPk",分片键必需为主键、int整数型,此键可以不包含在所传输字段中;

3.报错:

Exception in thread "taskGroup-0" com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-07], Description:[读取数据库数据失败. 请检查您的配置的 column/table/where/querySql或者向 DBA 寻求帮助.]. - 执行的SQL为: select WorkOrderId,CustStoreId,CustStoreName,CustStoreCode,PreCustStoreId,PreCustStoreName,CustSettleId,CustSettleName,IsCustomer,CustCoopType,ProCode,AppCode,ProName,CityCode,CityName,AreaCode,AreaName,InstallAddress,InstallTime,RequiredTime,LinkMan,LinkTel,SourceType,SecondLinkTel,SecondLinkMan,WarehouseId,WarehouseName,Remark,IsUrgent,CustUniqueSign,CreatePersonCode,CreatePersonName,EffectiveTime,OrderType,EffectiveSuccessfulTime,CreatedById,CreatedAt,UpdatedById,UpdatedAt,DeletedById,DeletedAt,Deleted,AppointStatus,ServiceCode,CreateType,ServiceName,RecCompleteTime,RecRecordPersonCode,RecRecordPersonName,WfOrderCompletetime,WfAssignor,WfDirectOrderAccept,WorkOdrTypeStatus,WorkOdrCloseStatusName,WorkOdrSuspendStatus,WorkStatus,WorkOdrClosedAt,AFCCreatedAt,AFCRetreatTime,AFCFollowFrequency,AFCIfRemove,AFCReturnedTypeCode,VHSAppraisalStatus,VHSReceiverLoginName,OperatingInfo,OperatedInfo,ServiceProviderCode,WorkflowInfo,WorkServiceSubject,AFCWorkFinanceRecord,AFCPaymentReport,VHSWorkFileReceiveInfo,CustomerId,CustomerName from workjsoninfo where (169036 <= Id AND Id < 338071) 具体错误信息为:com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Application was streaming results when the connection failed. Consider raising value of 'net_write_timeout' on the server.

解决方法:

增加源端 wait_timeout 参数值。

NULL值处理

问题:mysql2mongo数据库传输,不识别mysql内josn字段,因为datax会给json增加转义字符 ""

解决方法:源端字段类型改为varchar,入目标为array类型,mongodbwriter需要指定特殊解析数组分隔符;

注意事项

1.单 executor执行速度有限,尽量开多管道;

本文作者:wangcw

本文链接:

版权声明:本博客所有文章除特别声明外,均采用 BY-NC-SA 许可协议。转载请注明出处!