基于Leap+Motion的采煤设备虚拟装配研究与实现.pdf

工程硕士专业学位论文 基基于于Leap Motion 的的采采煤煤设设备备虚虚拟拟装装配配研研究究与与实实现现 Research and Implementation of Virtual Assembly of Coal Mining Equipment based on Leap Motion 作作者者范烨博范烨博 导导师师彭延军彭延军教授教授 山东科技大学 二〇二〇年六月 万方数据 中图分类号 TN911.7学校代码10424 UDC004密级公开 山东科技大学 工程硕士专业学位论文 基基于于Leap Motion 的的采采煤煤设设备备虚虚拟拟装装配配研研究究与与实实现现 Research and implementation of virtual assembly of coal mining equipment based on Leap Motion 作者范烨博入学时间2017 年9 月 导师彭延军职称教授 申请学位工程硕士所在学院计算机科学与工程 学科（类别） 工程方向（领域） 计算机技术 答辩日期2020年6月10 日 提交日期2020年6月15日 万方数据 万方数据 万方数据 万方数据 摘要摘要 在采煤设备的制造以及装配领域，由于传统制造与装配方式存在的大型机械移动不 便、装配成本高、装配训练受时间地点等种种约束，已经越来越无法满足人们的需求，也 增加了采煤机、掘运机等设备新产品的设计和研发过程中的困难。 近些年来，虚拟现实技术的快速发展，令其已经成为计算机视觉和人机交互等研究领 域中的研究主向，与此同时，虚拟装配技术作为 VR 技术的衍生品，也逐渐成为虚拟现实 技术的重要组成部分，在虚拟现实技术和虚拟仿真领域得到了广泛的研究和应用。虚拟装 配技术可以确定采煤设备在装配的过程中装配、拆卸以及重组的最优次序，还能对整体的 装配过程进行验证和修改，并分析装配的可用性、可造性和可维护性。国内相应的采煤机、 掘运机等设备的装配和研发主要是在学习和模仿国外的先进技术，在此情况下，先进的虚 拟装配与场景仿真技术开发出一个更实用、更高效的采煤设备的虚拟装配方法就显得更加 必要。 本文基于局域网内协同技术，利用 Leap Motion 作为主要硬件设备，对采煤设备进行 多人协同手势控制的虚拟装配。首先导入三维模型，基于 Leap Motion 传感器，获取手势 数据，精确计算手部移动速度、位置和方向进行手势识别；然后通过光线追踪、进行碰撞 检测，若有碰撞则对模型变换坐标进行旋转，实现装配部件的拾取及装配；再通过局域网 内协同技术，利用 Unity 进行多用户局域网互联，将虚拟试验场景网络化，在多用户手势 控制下，进行操作者与装配部件的交互，实现装配部件的拆分与装配。 本文提出的方法致力于解决对采煤机、掘运机等采煤设备的设计研发过程中的装配困 难问题，可以实现在对采煤设备进行装配过程中的多人手势协同控制，拓宽了操作者的操 作空间，让虚拟装配工作不再仅仅局限于一个操作者的操作空间。 关键词关键词多人协同；Leap Motion；虚拟装配技术；Unity 3D；手势交互 万方数据 ABSTRACT In the field of coal mining equipment manufacturing and assembly, because traditional manufacturing and assembly of large mechanical mobile inconvenience and high cost of assembly, assembly training site constraints such as time, already more and more cant satisfy peoples needs, has increased the coal winning machine, digging machine equipment such as the difficulties in the process of the design and development of new products. In recent years, the rapid development of virtual reality technology, make its have become main research direction in the field of computer vision and human-computer interaction, virtual assembly technology derived from virtual reality technology has become an important part of virtual reality technology, in the field of virtual reality technology and virtual simulation has been widely research and application.Virtual assembly technology can determine the optimal sequence of assembling, disassembling and reassembling coal mining equipment in the process ofassembly, verifyandmodifytheassemblyprocess,andanalyzetheavailability, manufacturability and maintainability of the assembly.In this case, it is more necessary for advanced virtual assembly and scene simulation technology to develop a more practical and efficient virtual assembly for coal mining equipment. Based on the technology of LAN collaboration, this paper uses Leap Motion as the main hardware device to carry out virtual assembly of coal mining equipment through multi-person collaborative gesture control. Firstly, the 3d model is imported, based on Leap Motion sensor, obtain gesture data, and accurately calculate the moving speed, position and direction of the hand for gesture recognition; Then, to take the ollision detection by ray tracing. If there is any collision, the model transation coordinates are rotated to realize the pick up and assembly of assembly parts; Finally, through in the technology of LAN, the use of Unity for multi-user LAN interconnection, and networking the virtual experiment scene.Under the multi-user gesture control, the interaction between the operators hand and the assembly part in the virtual scene is realized, and the assembly part is split and assembled. This paper puts forward the s to solve the coal winning machine, digging machine and other mining equipment design assembly difficult problem in the process of research and development, can be implemented in the assembly process of coal mining equipment more than gestures collaborative control, broaden the space of operators, virtual assembly job is no longer just limited to a space of the operators. Key words Multi-person collaboration; Leap Motion; Virtual assembly technology; Unity 3D; Hand interaction 万方数据 目录目录 图清单......................................................................................................................................I 表清单.....................................................................................................................................II 1 绪论..................................................................................................................................... 1 1.1 研究背景及意义...................................................................................................... 1 1.1.1 研究背景....................................................................................................... 1 1.1.2 研究目的及意义........................................................................................... 2 1.2 国内外研究现状...................................................................................................... 2 1.2.1 虚拟现实技术在煤矿领域的研究现状....................................................... 2 1.2.2 虚拟装配的研究现状................................................................................... 4 1.3 论文的主要研究内容.............................................................................................. 4 1.4 论文章节安排.......................................................................................................... 5 2 虚拟交互技术与设备......................................................................................................... 7 2.1 虚拟交互技术.......................................................................................................... 7 2.2 虚拟交互设备.......................................................................................................... 7 2.2.1 Leap Motion 手势识别设备...........................................................................8 2.2.2 Kinect 光学设备.............................................................................................9 2.2.3 头戴式显示器............................................................................................... 9 2.3 本章小结.................................................................................................................. 9 3 虚拟场景中的手势识别................................................................................................... 11 3.1 手势数据处理........................................................................................................ 11 3.1.1 坐标转换..................................................................................................... 11 3.1.2 Leap Motion 的数据获取.............................................................................12 3.2 手势特征提取........................................................................................................ 13 3.3 手势识别................................................................................................................ 14 3.3.1 手势识别算法............................................................................................. 14 3.3.2 辅助交互设备............................................................................................. 16 3.3.3 实验结果及分析......................................................................................... 16 3.4 应用........................................................................................................................ 18 3.5 本章小结................................................................................................................ 19 4 采煤设备虚拟装配系统................................................................................................... 20 4.1 系统功能模块........................................................................................................ 20 4.2 虚拟协同装配的总体设计.................................................................................... 21 4.3 装配交互................................................................................................................ 22 4.3.1 交互时的视觉反馈机制............................................................................. 22 万方数据 4.3.2 碰撞检测..................................................................................................... 24 4.3.3 装配部件的坐标变换................................................................................. 27 4.3.4 手与装配部件的交互................................................................................. 28 4.3.5 装配部件之间的交互................................................................................. 30 4.3.6 装配交互效果展示..................................................................................... 30 4.4 协同操作................................................................................................................ 32 4.4.1 协同架构设计............................................................................................. 32 4.4.2 基于 TCP 协议的网络通信........................................................................33 4.4.3 虚拟实验场景网络化................................................................................. 34 4.4.4 虚拟场景的实时同步................................................................................. 35 4.5 系统应用................................................................................................................ 36 4.6 本章小结................................................................................................................ 40 5 总结与展望....................................................................................................................... 41 5.1 总结........................................................................................................................ 41 5.2 展望........................................................................................................................ 41 参考文献............................................................................................................................... 42 作者简历............................................................................................................................... 46 致谢....................................................................................................................................... 47 万方数据 Contents List of figures........................................................................................................................... I List of Tables...........................................................................................................................II 1 Introduction...........................................................................................................................1 1.1Research background and significance.......................................................................1 1.1.1 Research background......................................................................................1 1.1.2 Research purpose and significance.................................................................2 1.2 Research Actuality at domestic and overseas............................................................2 1.2.1 Research status of virtual reality technology in coal field..............................2 1.2.2 Research status of virtual assembly................................................................4 1.3 The main research content of the paper.....................................................................4 1.4 Chapter arrangement of the paper..............................................................................5 2 Research on key technologies of the system........................................................................ 7 2.1 Virtual interaction technology...................................................................................7 2.2 Virtual interactive device...........................................................................................7 2.2.1 Leap Motion gesture recognition device........................................................ 8 2.2.2 Kinect optical device...................................................................................... 9 2.2.3 Head-mounted display....................................................................................9 2.3 The summary of this chapter..................................................................................... 9 3 Gesture recognition in a virtual scene.................................................................................11 3.1 Gesture data processing...........................................................................................11 3.1.1 Coordinate transation............................................................................11 3.1.2 Leap Motions data acquisition.....................................................................12 3.2 Feature extraction of gesture....................................................................................13 3.3 Gesture recognition..................................................................................................14 3.3.1 Gesture recognition algorithm......................................................................14 3.3.2 Auxiliary interactive device..........................................................................16 3.3.3 Experimental results and analysis.................................................................16 3.4 Implementation........................................................................................................18 3.5 The summary of this chapter................................................................................... 19 4 Coal mining equipment virtual assembly system...............................................................20 4.1 System module structure..........................................................................................20 4.2 Overall design of virtual collaborative assembly.................................................... 21 4.3 Assembly interaction............................................................................................... 22 万方数据 4.3.1 Visual feedback mechanism in interaction...................................................22 4.3.2 Collision detection........................................................................................24 4.3.3 Coordinate transation of assembly parts...............................................27 4.3.4 Hand to assembly interaction........................................................................28 4.3.5 Interactions between assembly component...................................................30 4.3.6 Assembly interactive effect display..............................................................30 4.4 Cooperative operation..............................................................................................32 4.4.1 Collaborative architecture design................................................................. 32 4.4.2 Network communication based on TCP.......................................................33 4.4.3 The virtual experiment scene is networked..............................