-
Notifications
You must be signed in to change notification settings - Fork 1
/
faq.html
511 lines (487 loc) · 18.8 KB
/
faq.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta content="Apache Forrest" name="Generator">
<meta name="Forrest-version" content="0.9">
<meta name="Forrest-skin-name" content="pelt">
<title>Scriptella ETL Frequently Asked Questions</title>
<link type="text/css" href="skin/basic.css" rel="stylesheet">
<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
<link type="text/css" href="skin/profile.css" rel="stylesheet">
<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
<link rel="shortcut icon" href="favicon.ico">
</head>
<body onload="init()">
<script type="text/javascript">ndeSetTextSize();</script>
<div id="top">
<!--+
|header
+-->
<div class="header">
<!--+
|start group logo
+-->
<div class="grouplogo">
<a href="http://scriptella.org"></a>
</div>
<!--+
|end group logo
+-->
<!--+
|start Project Logo
+-->
<div class="projectlogo">
<a href="http://scriptella.org/"><img class="logoImage" alt="Scriptella" src="images/site-logo.png" title="Open Source ETL and Script Execution Tool."></a>
</div>
<!--+
|end Project Logo
+-->
<!--+
|start Search
+-->
<div class="searchbox">
<form action="http://www.google.com/search" method="get" class="roundtopsmall">
<input value="scriptella.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">
<input name="Search" value="Search" type="submit">
</form>
</div>
<!--+
|end search
+-->
<!--+
|start Tabs
+-->
<ul id="tabs">
<li class="current">
<a class="selected" href="index.html">Home</a>
</li>
<li>
<a class="unselected" href="reference/index.html">Reference</a>
</li>
<li>
<a class="unselected" href="howto/migrate-from-ant.html">How-Tos</a>
</li>
</ul>
<!--+
|end Tabs
+-->
</div>
</div>
<div id="main">
<div id="publishedStrip">
<!--+
|start Subtabs
+-->
<div id="level2tabs"></div>
<!--+
|end Endtabs
+-->
<script type="text/javascript"><!--
document.write("Last Published: " + document.lastModified);
// --></script>
</div>
<!--+
|breadtrail
+-->
<div class="breadtrail">
<script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
</div>
<!--+
|start Menu, mainarea
+-->
<!--+
|start Menu
+-->
<div id="menu">
<div onclick="SwitchMenu('menu_1.1', 'skin/')" id="menu_1.1Title" class="menutitle">Scriptella</div>
<div id="menu_1.1" class="menuitemgroup">
<div class="menuitem">
<a href="index.html" title="About Scriptella">About</a>
</div>
<div class="menuitem">
<a href="license.html" title="Scriptella License">License</a>
</div>
<div class="menuitem">
<a href="changes.html">Change History</a>
</div>
<div class="menuitem">
<a href="support.html" title="Scriptella Project Support">Support</a>
</div>
<div class="menuitem">
<a href="download.html" title="Download Scriptella Binary and Source Distribution">Download</a>
</div>
<div class="menuitem">
<a href="links.html" title="Scriptella Related Links">Links</a>
</div>
</div>
<div onclick="SwitchMenu('menu_selected_1.2', 'skin/')" id="menu_selected_1.2Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Getting Started</div>
<div id="menu_selected_1.2" class="selectedmenuitemgroup" style="display: block;">
<div class="menuitem">
<a href="tutorial.html" title="Two Minute Tutorial">Two Minute Tutorial</a>
</div>
<div class="menupage">
<div class="menupagetitle">FAQ</div>
</div>
</div>
<div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">Documentation</div>
<div id="menu_1.3" class="menuitemgroup">
<div class="menuitem">
<a href="reference/index.html" title="Scriptella Reference Documentation">Reference Documentation</a>
</div>
<div class="menuitem">
<a href="reference/drivers.html">Drivers Matrix</a>
</div>
<div class="menuitem">
<a href="docs/api/index.html">API Docs</a>
</div>
<div class="menuitem">
<a href="docs/dtd/index.html">DTD Reference</a>
</div>
<div onclick="SwitchMenu('menu_1.3.5', 'skin/')" id="menu_1.3.5Title" class="menutitle">How-Tos</div>
<div id="menu_1.3.5" class="menuitemgroup">
<div class="menuitem">
<a href="howto/migrate-from-ant.html">Migrate from Ant SQL Task</a>
</div>
<div class="menuitem">
<a href="howto/initialize-database.html">Automated Database Initialization</a>
</div>
</div>
</div>
<div id="credit">
<!-- TODO Additional Links/Tags Go Here -->
</div>
<div id="roundbottom">
<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
<!--+
|alternative credits
+-->
<div id="credit2"></div>
</div>
<!--+
|end Menu
+-->
<!--+
|start content
+-->
<div id="content">
<div title="Portable Document Format" class="pdflink">
<a class="dida" href="faq.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
PDF</a>
</div>
<div class="trail">Font size:
<input value="Reset" class="resetfont" title="Reset text" onclick="ndeSetTextSize('reset'); return false;" type="button">
<input value="-a" class="smallerfont" title="Shrink text" onclick="ndeSetTextSize('decr'); return false;" type="button">
<input value="+a" class="biggerfont" title="Enlarge text" onclick="ndeSetTextSize('incr'); return false;" type="button">
</div>
<h1>Scriptella ETL Frequently Asked Questions</h1>
<div id="front-matter">
<div id="minitoc-area">
<ul class="minitoc">
<li>
<a href="#part-N1000C">1. General</a>
<ul class="minitoc">
<li>
<a href="#DRIVERS">1.1. Where can I find documentation on script syntax for driver FOO/BAR/...?</a>
</li>
<li>
<a href="#ODBC">1.2. Does Scriptella support Microsoft Access?</a>
</li>
<li>
<a href="#BIND_VARIABLES">1.3. What is the difference between $variable / ${expression} and ?variable / ?{expression} syntax.</a>
</li>
<li>
<a href="#SET_VARIABLE">1.4. How to share a mutable variable between script/query elements of an ETL file?</a>
</li>
<li>
<a href="#EXECUTE_STATIC_METHOD">1.5. How to call a static method in JEXL expression?</a>
</li>
<li>
<a href="#COLUMN_SPACES">1.6. How to escape spaces in field names?</a>
</li>
</ul>
</li>
<li>
<a href="#part-N100A2">2. SQL</a>
<ul class="minitoc">
<li>
<a href="#PLSQL">2.1. How to create an Oracle trigger (run a PL/SQL block)?</a>
</li>
<li>
<a href="#COMMIT">2.2. How to control commit options and transactions?</a>
</li>
<li>
<a href="#UPSERT">2.3. How to UPSERT (update or insert into a table?)</a>
</li>
</ul>
</li>
<li>
<a href="#part-N10128">3. CSV/Text</a>
<ul class="minitoc">
<li>
<a href="#NULLS">3.1. How to output null value as empty string in a CSV/Text file.</a>
</li>
</ul>
</li>
</ul>
</div>
</div>
<a name="part-N1000C"></a>
<h2 class="boxed">1. General</h2>
<div class="section">
<a name="DRIVERS"></a>
<h3 class="boxed">1.1. Where can I find documentation on script syntax for driver FOO/BAR/...?</h3>
<p>An up to date drivers documentation is available in the <a href="docs/api/">Scriptella Javadoc</a>.
<a href="reference/index.html">Reference Documentation</a> also covers drivers usage and scripts syntax.</p>
<a name="ODBC"></a>
<h3 class="boxed">1.2. Does Scriptella support Microsoft Access?</h3>
<p>Microsoft Access and other databases with ODBC interface are supported via ODBC-JDBC bridge driver which comes with Sun's JRE.
Download <a href="download.html">ODBC example</a> for more details. Connection declaration example:</p>
<pre class="code">
<connection url="jdbc:odbc:DRIVER={Microsoft Access Driver (*.mdb)};DBQ=Northwind.mdb"/>
</pre>
<a name="BIND_VARIABLES"></a>
<h3 class="boxed">1.3. What is the difference between $variable / ${expression} and ?variable / ?{expression} syntax.</h3>
<p>Binding variables syntax varies between drivers. JDBC drivers use the following rules
for properties substitution:</p>
<ul>
<li>$variable - inserts a value of the variable as text content.</li>
<li>${expression} - braces are used for JEXL expressions. For example
${column1+column2} inserts a sum of 2 columns.</li>
<li>?variable, ?{expression} - syntax is the same as in 2 previous examples, but the
result of evaluation is set as a prepared statement parameter, thus increasing the performance
and eliminating the need to escape values.
<br>Please note that currently ?{} syntax is only supported by the JDBC drivers.</li>
</ul>
<p>See <a href="reference/index.html#BIND_VARIABLES">Reference Manual</a> for additional details.</p>
<a name="SET_VARIABLE"></a>
<h3 class="boxed">1.4. How to share a mutable variable between script/query elements of an ETL file?</h3>
<p>In general such practice in not recommended because often it's a sign of a bad design.
It's like using GOTOs or global variables in programming languages.
But in several cases using a global variable may help to achieve required goal. We propose 2 approaches:</p>
<ol>
<li>Use etl.globals map to work with global variables. Example ${etl.globals['globalVar']}</li>
<li>Another approach is similar to the technique utilized in anonymous inner classes to
modify a single-element array declared as a final variable.</li>
</ol>
<p>The following example demonstrates both approaches:</p>
<pre class="code">
<etl>
<connection driver="script" id="js"/>
<connection url="jdbc:...." id="db"/>
<!-- Set number of records as a global variable -->
<!-- Note that JEXL syntax etl.globals['globalVar'] does not work in JavaScript -->
<query connection-id="db">
select count(id) as c from Errors
<script connection-id="js">
etl.globals.put('errorsCount', c);
<script>
</query>
<!-- Then reuse this variable in other parts of ETL file -->
<script connection-id="js" if="etl.globals.get('errorsCount') gt 0">
java.lang.System.out.println('errors count ='+etl.globals.get('errorsCount'));
</script>
<!-- Alternatively an outer query can be used to share an array based variable between scripts -->
<query connection-id="js">
var pseudoGlobalVar = []; //Declare pseudo-global variable available to nested element
query.next(); //Executes child scripts
<script> //Updates the variable
pseudoGlobalVar[0] = 1;
</script>
<script> //Outputs updated value of global variable
java.lang.System.out.println('pseudoGlobalVar[0]='+pseudoGlobalVar[0]);
</script>
</query>
</etl>
</pre>
<a name="EXECUTE_STATIC_METHOD"></a>
<h3 class="boxed">1.5. How to call a static method in JEXL expression?</h3>
<p>Load a class by name by using class:forName Scriptella function</p>
<p>Example. Call System.getProperty:</p>
<pre class="code">
${class:forName('java.lang.System').getProperty('propName')}
</pre>
<a name="COLUMN_SPACES"></a>
<h3 class="boxed">1.6. How to escape spaces in field names?</h3>
<p>You can use <a href="docs/api/scriptella/core/EtlVariable.html"><span class="codefrag">etl context variable</span></a> ${etl.getParameter('var name')} to reference any column names.
Additionally several drivers
including CSV and JDBC allow referencing columns by an index, i.e. $1, $2 ... $n.
<br>Example:
</p>
<pre class="code">
<query connection-id="csv"> <!-- Read CSV file content -->
<script connection-id="text">
<!-- Print columns, column 2,3 are referenced by name, 1,4 - by index -->
$1,$secondColumn,${etl.getParameter('My Column')},$4
</script>
</query>
</pre>
</div>
<a name="part-N100A2"></a>
<h2 class="boxed">2. SQL</h2>
<div class="section">
<a name="PLSQL"></a>
<h3 class="boxed">2.1. How to create an Oracle trigger (run a PL/SQL block)?</h3>
<p>
To recognize Oracle PL/SQL statement blocks you'd have to specify <span class="codefrag">plsql=true</span> connection property
(supported only by <a href="reference/drivers.html#oracle">Scriptella Adapter for Oracle</a>).
In this case a slash(/) on a single line is used as a statement separator:</p>
<pre class="code">
<connection driver="oracle" ...>
plsql=true
</connection>
<script>
CREATE OR REPLACE TRIGGER secure_del_trigger
BEFORE DELETE
ON emp
FOR EACH ROW
DECLARE
unauthorized_deletion EXCEPTION;
BEGIN
IF <your business rule is violated> THEN
RAISE unauthorized_deletion;
END IF;
EXCEPTION
WHEN unauthorized_deletion
THEN
raise_application_error (-20500,
'This record cannot be deleted');
END;
/
-- Other statements separated with a slash on a single line
</script>
</pre>
<p>If you are using Oracle JDBC driver directly then set the following configuration properties:</p>
<pre class="code">
<connection driver="oracle.jdbc.driver.OracleDriver" ...>
statement.separator=/
statement.separator.singleline=true
</connection>
</pre>
<p>This idea is similar to <a href="http://www.javaddicts.net/blog/index.php/2005/06/13/executing-oracle-plsql-from-ant/">Ant solution</a>.</p>
<a name="COMMIT"></a>
<h3 class="boxed">2.2. How to control commit options and transactions?</h3>
<p>The following connection parameters are supported by the JDBC bridge:</p>
<ul>
<li>
<span class="codefrag">transaction.isolation</span> - Transaction isolation level name</li>
<li>
<span class="codefrag">autocommit</span> - Enables/disables auto-commit mode.</li>
<li>
<span class="codefrag">autocommit.size</span> - Enables/disables auto-commit mode.</li>
</ul>
<div class="note">
<div class="label">Note</div>
<div class="content">
In general avoid using <span class="codefrag">autocommit.size</span>, because in this case an ETL process cannot be rolled back correctly. Use this parameter only for performance critical operations (bulk inserts etc.).<br>
If the <span class="codefrag">autocommit</span> is true, then setting <span class="codefrag">autocommit.size</span> has no effect.
</div>
</div>
<p>
<strong>Example:</strong>
</p>
<pre class="code">
<connection driver="auto" url="jdbc:hsqldb:mem:test">
autocommit.size=4 <!-- Automatically commit after every 4th statement -->
transaction.isolation=SERIALIZABLE <!-- Sets TX level to Serializable -->
</connection></pre>
<a name="UPSERT"></a>
<h3 class="boxed">2.3. How to UPSERT (update or insert into a table?)</h3>
<p>You can leverage database-specific SQL statements like MERGE or UPSERT. See answers on StackOverflow:</p>
<ul>
<li>
<a href="http://stackoverflow.com/questions/237327/oracle-how-to-upsert-update-or-insert-into-a-table">Oracle</a>
</li>
<li>
<a href="http://stackoverflow.com/questions/1218905/how-do-i-update-if-exists-insert-if-not-aka-upsert-or-merge-in-mysql">MySQL</a>
</li>
<li>
<a href="http://stackoverflow.com/questions/108403/solutions-for-insert-or-update-on-sql-server">MS SQL Server</a>
</li>
</ul>
<p>An alternative approach, which is universal but is typically slower, is to use count(*) statement and do insert or update accordingly:</p>
<pre class="code">
<query connection-id="in">
SELECT * FROM Persons_In
<query connection-id="out">
SELECT COUNT(*) as cnt FROM Person_Out WHERE Person_ID=?Person_ID
<!-- If nothing found - insert -->
<script if="cnt==0">
INSERT INTO Person_OUT VALUES (Person_ID, Person_Name, ...);
</script>
<!-- Otherwise - update -->
<script if="cnt gt 0">
UPDATE Person_OUT SET Person_Name=?Person_Name WHERE Person_ID = ?Person_ID;
</script>
</query>
</query>
</pre>
</div>
<a name="part-N10128"></a>
<h2 class="boxed">3. CSV/Text</h2>
<div class="section">
<a name="NULLS"></a>
<h3 class="boxed">3.1. How to output null value as empty string in a CSV/Text file.</h3>
<p>Suppose we have 3 variables - a='valueA';b=null and c='valueC'.</p>
<p>Here is a <a href="docs/api/scriptella/driver/csv/package-summary.html#package_description">CSV</a>
script which outputs them in a single line:</p>
<pre class="code">
$a,$b,$c
</pre>
<p>The output for this script:</p>
<pre class="code">
valueA,<strong>$b</strong>,valueC
</pre>
<p>
By default NULL variables are not substituted, because Scriptella substitution engine cannot
distinguish null value from undeclared variable. This behaviour is not always desired. As a workaround you can specify a null_string connection property as follows:</p>
<pre class="code">
<connection driver="text">
null_string=<!-- Expand nulls to empty string -->
</connection>
</pre>
<p>And the output is:</p>
<pre class="code">valueA,,valueC</pre>
</div>
</div>
<!--+
|end content
+-->
<div class="clearboth"> </div>
</div>
<div id="footer">
<!--+
|start bottomstrip
+-->
<div class="lastmodified">
<script type="text/javascript"><!--
document.write("Last Published: " + document.lastModified);
// --></script>
</div>
<div class="copyright">
Copyright ©
2006-2019 <a href="http://scriptella.org/license.html">The Scriptella Project Team.</a>
</div>
<div id="feedback">
Send feedback about the website to:
<a id="feedbackto" href="mailto:[email protected]?subject=Feedback%C2%A0faq.html">Fyodor Kupolov</a>
</div>
<!--+
|end bottomstrip
+-->
</div>
<!-- Start of StatCounter Code for Default Guide -->
<script type="text/javascript">
var sc_project=10775960;
var sc_invisible=1;
var sc_security="53eaed1c";
var scJsHost = (("https:" == document.location.protocol) ?
"https://secure." : "http://www.");
document.write("<sc"+"ript type='text/javascript' src='" +
scJsHost+
"statcounter.com/counter/counter.js'></"+"script>");
</script>
<!-- End of StatCounter Code for Default Guide -->
</body>
</html>