-
Notifications
You must be signed in to change notification settings - Fork 0
/
VOICE.html
120 lines (103 loc) · 5.53 KB
/
VOICE.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
<!DOCTYPE HTML>
<!--
Twenty by HTML5 UP
html5up.net | @ajlkn
Free for personal and commercial use under the CCA 3.0 license (html5up.net/license)
-->
<html>
<head>
<title>Nanovis Group</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" />
<link rel="stylesheet" href="assets/css/main.css" />
<noscript><link rel="stylesheet" href="assets/css/noscript.css" /></noscript>
</head>
<body class="index is-preload">
<div id="page-wrapper">
<!-- Header -->
<div id="header-container"></div>
<!-- Main -->
<article id="main">
<header class="special container">
<h2><strong>VOICE: Visual Oracle for Interaction, Conversation, and Explanation</strong></h2>
</header>
<!-- Content -->
<section id="pub_1" class="wrapper style4 container special-alt">
<div class="row gtr-50">
<div class="col-12 col-12-narrower">
<div class="image fit">
<img src="./images/VOICE.png" />
<p class="fig-text"></p>
</div>
</div>
</div>
<div class="row gtr-50">
<div class="col-12 col-12-narrower">
<header>
<h2>Authors</h2>
<a href="https://cemse.kaust.edu.sa/people/person/donggang-jia">Donggang Jia</a>,
<a href="https://cemse.kaust.edu.sa/vcc/people/person/alexandra-irger">Alexandra Irger</a>,
<a href="https://liu.se/en/employee/lonbe19">Lonni Besançon</a>,
<a href="https://cemse.kaust.edu.sa/cs/people/person/ondrej-strnad">Ondrej Strnad</a>,
<a href="https://cemse.kaust.edu.sa/people/person/deng-luo">Deng Luo</a>,
<a href="https://www.umu.se/en/staff/johanna-bjorklund/">Johanna Björklund</a>,
<a href="https://liu.se/en/employee/andyn27">Anders Ynnerman</a>,
<a href="https://cemse.kaust.edu.sa/nanovis/people/person/ivan-viola">Ivan Viola</a>
</header>
<h2>Description</h2>
<p>
We present VOICE, a novel approach to science communication that connects large language models' (LLM) conversational capabilities with interactive exploratory visualization. VOICE introduces several innovative technical contributions that drive our conversational visualization framework. Our foundation is a pack-of-bots that can perform specific tasks, such as assigning tasks, extracting instructions, and generating coherent content. We employ fine-tuning and prompt engineering techniques to tailor bots' performance to their specific roles and accurately respond to user queries. Our interactive text-to-visualization method generates a flythrough sequence matching the content explanation. Besides, natural language interaction provides capabilities to navigate and manipulate the 3D models in real-time. The VOICE framework can receive arbitrary voice commands from the user and respond verbally, tightly coupled with corresponding visual representation with low latency and high accuracy. We demonstrate the effectiveness of our approach by applying it to the molecular visualization domain: analyzing three 3D molecular models with multi-scale and multi-instance attributes. We finally evaluate VOICE with the identified educational experts to show the potential of our approach.
</p>
<h2>Sources</h2>
<ul>
<li><a href="https://osf.io/g7fbr">Supplymental Materials</a></li>
</ul>
<p>
</p>
<footer>
<h2>Publications</h2>
<p>
Donggang Jia, Alexandra Irger, Lonni Besançon, Ondrej Strnad, Deng Luo, Johanna Björklund, Anders Ynnerman, Ivan Viola.
<a href="https://arxiv.org/abs/2304.04083">VOICE: Visual Oracle for Interaction, Conversation, and Explanation</a>
<em>, preprint, 2024.</em>
<span id="bibtex" onclick="$('#bibtex-text').slideToggle(500)"><strong>[BibTeX]</strong></span>
</p>
<code id="bibtex-text" style="display: none;">
@misc{jia2024voice,
title={VOICE: Visual Oracle for Interaction, Conversation, and Explanation},
author={Donggang Jia and Alexandra Irger and Lonni Besancon and Ondrej Strnad and Deng Luo and Johanna Bjorklund and Anders Ynnerman and Ivan Viola},
year={2024},
eprint={2304.04083},
archivePrefix={arXiv},
primaryClass={cs.HC}
}
</code>
</p>
</footer>
</div>
</div>
</section>
</article>
<!-- Footer -->
<footer id="footer">
<ul class="icons">
<li><a href="https://github.com/nanovis" class="icon brands circle fa-github"><span class="label">Github</span></a></li>
<li><a href="https://twitter.com/nanovis_group" class="icon brands circle fa-twitter"><span class="label">Twitter</span></a></li>
</ul>
<ul class="copyright">
<li>© Nanovis</li><li>Design: <a href="http://html5up.net">HTML5 UP</a></li>
</ul>
</footer>
</div>
<!-- Scripts -->
<script src="assets/js/jquery.min.js"></script>
<script src="assets/js/jquery.dropotron.min.js"></script>
<script src="assets/js/jquery.scrolly.min.js"></script>
<script src="assets/js/jquery.scrollex.min.js"></script>
<script src="assets/js/browser.min.js"></script>
<script src="assets/js/breakpoints.min.js"></script>
<script src="assets/js/util.js"></script>
<script src="assets/js/main.js"></script>
<script src="assets/js/load_header.js"></script>
</body>
</html>