prep for new release

caltechlibrary · Nov 22, 2017 · 3e933c1 · 3e933c1
1 parent a394e62
commit 3e933c1
Show file tree

Hide file tree

Showing 54 changed files with 381 additions and 226 deletions.
diff --git a/README.md b/README.md
@@ -17,6 +17,7 @@ general purpose shell scripting.
 + [jsonjoin](docs/jsonjoin.html) - a tool for joining JSON object documents
 + [jsonmunge](docs/jsonmunge.html) - a tool to transform JSON documents into something else
 + [jsonrange](docs/jsonrange.html) - a tool for iterating for JSON maps and arrays
++ [splitstring](docs/splitstring.html) - splits a string using a delimiting string and returns a JSON array
 + [vcard2json](docs/vcard2json.html) - an experimental tool to convert vCards to JSON
 + [xlsx2csv](docs/xlsx2csv.html) - a tool for converting Excel Workbooks sheets to a CSV file(s)
 + [xlsx2json](docs/xlsx2json.html) - a tool for converting Excel Workbooks to JSON files

diff --git a/TODO.html b/TODO.html
@@ -0,0 +1,133 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Caltech Library's Digital Library Development Sandbox</title>
+    <link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'>
+    <link rel="stylesheet" href="/css/site.css">
+</head>
+<body>
+<header>
+<a href="http://library.caltech.edu"><img src="/assets/liblogo.gif" alt="Caltech Library logo"></a>
+</header>
+<nav>
+<ul>
+<li><a href="/">Home</a></li>
+<li><a href="./">README</a></li>
+<li><a href="license.html">LICENSE</a></li>
+<li><a href="install.html">INSTALL</a></li>
+<li><a href="docs/">Documentation</a></li>
+<li><a href="how-to/">How To</a></li>
+<li><a href="https://github.com/caltechlibrary/datatools">Github</a></li>
+</ul>
+
+</nav>
+
+<section>
+<h1>Action Items</h1>
+
+<h2>Bug</h2>
+
+<h2>Next</h2>
+
+<ul>
+<li>[ ] csvrows would output a range of rows (e.g. [2:] would be all rows but the first row)</li>
+<li>[ ] csv utilities to support integer ranges notation for columns and rows references, E.g. &ldquo;1,3:4,7,10:&rdquo; or all</li>
+</ul>
+
+<h2>Someday, Maybe</h2>
+
+<ul>
+<li>[ ] finddir should have an option to exclude directories (e.g. exclude .git directories from a listing)</li>
+<li>[ ] textscraper - a tool for select out text and storing it as a JSON field value, sort grep plus sed cleanup and semi-structured text (e.g. webpage)
+
+<ul>
+<li>look at how cut, sed, grep are commonly used in my scripts and merge that functionality into a single tool</li>
+</ul></li>
+<li>[ ] csvcols, csvrows should have a length option to give you a number of columns or rows respectively</li>
+<li>[ ] csvcols, csvrows should have a filter option to filter to support filting output conditionally</li>
+<li>[ ] csvsort should allow a multi-column sort respecting column headings
+
+<ul>
+<li>plus column number would be ascending by that column</li>
+<li>minos column number would be descending by that column</li>
+<li>sort would be read from left to right</li>
+<li>it would be good to include support for column names and not just column numbers to describe the sort</li>
+</ul></li>
+<li>[ ] jsonmodify takes a JSON document, a dotpath and value then creates/updates the dotpath in the JSON document with the new value
+
+<ul>
+<li>&rdquo;(delete DOTPATH)&rdquo; would remove the property described by the dotpath</li>
+<li>&rdquo;(update DOTPATH NEW_VALUE)&rdquo; would replace the property described by the dotpath with a new value (value can be a string, number, or JSON)</li>
+<li>&rdquo;(create&rdquo; DOTPATH NEW_VALUE)&rdquo; would add a new property at the described dotpath with a new value (value can be a string, number, or JSON)</li>
+<li>&rdquo;(join DOTH_PATH SEP)&rdquo; combines JSON array elements into a string version using separator</li>
+<li>&rdquo;(concat DOTPATH1 DOTPATH2&hellip; SEP)&rdquo; combines values into a concatenated string, it takes one or more dotpath values (must be string or number) and return them as a concatenated value (concat .last_name .first_name &ldquo;, &ldquo;) would return a last name comma first name string.</li>
+<li>&rdquo;(split DOTH_PATH SEP)&rdquo; turns a string into an array of strings using separator</li>
+</ul></li>
+<li>[ ] csvcols, csvrows should have a filter mechanism should provide a mechanism to filter by column or row
+
+<ul>
+<li>using a prefix notation (e.g. &lsquo;(and (eq (join (cols (colNo &ldquo;Last Name&rdquo;) (colNo &ldquo;First Name&rdquo;)) &ldquo;, &ldquo;) &ldquo;Doiel, R. S.&rdquo;) (gt (cols 4) &ldquo;2017-06-12&rdquo;))&rsquo;)</li>
+</ul></li>
+<li>[ ] csvfind, csvjoin should have an inverted match operation</li>
+<li>[ ] a range should accept the word &ldquo;all&rdquo; as well as comma delimited list of rows and ranges</li>
+<li>[ ] Add -uuid and -skip-header-row options constistantly to all csv tools
+
+<ul>
+<li>[ ] csvcols</li>
+</ul></li>
+<li>[ ] unify the options vocabulary to work the same between each cli
+
+<ul>
+<li>Need a common approach to column ranges in csvcols, csvfind, csvjoin</li>
+<li>csv2json, csv2mdtable, csv2xlsx should accept a column and row range option for output</li>
+</ul></li>
+<li>[ ] csvfind add filter by row number (helpful when combined with csvcols for snapshotting the middle of a table)</li>
+<li>[ ] csv2json should have an option that will include a row number in JSON blob output</li>
+<li>[ ] csv2json should have the options to normalize property names in JSON objects
+
+<ul>
+<li>camel case</li>
+<li>snake case</li>
+<li>lower case/upper case</li>
+<li>space to underscores</li>
+<li>strip punctuation</li>
+<li>rename keys</li>
+</ul></li>
+<li>[ ] csvrotate would take a CSV file as import and output columns as rows</li>
+<li>[ ] smartcat would function like cat but with support for ranges of lines (e.g. show me last 20 lines: smartcat -start=0 -end=&ldquo;-20&rdquo; file.txt; cat starting with 10th line: smartcat -start=10 file.txt)
+
+<ul>
+<li>[ ] allow prefix line number with a specific delimiter (E.g. comma would let you cat a CSV file adding row numbers as first column)</li>
+<li>[ ] show lines with prefix, suffix, containing or regxp</li>
+<li>[ ] show lines without prefix, suffix, containing or regexp</li>
+</ul></li>
+</ul>
+
+<h2>Completed</h2>
+
+<ul>
+<li>[x] csvcols -col option should not be a boolean, it should take a range like other csv cli</li>
+<li>[x] utilities should use starting index of 1 instead of zero as humans refer to column 1 when intending to work on the first column</li>
+<li>[x] for all cli the -delimiter option should support special characters like \t, \n, \r</li>
+<li>[x] csvfind would accept CSV input from stdin and output rows with matching column values
+
+<ul>
+<li>E.g. <code>cat file1.csv | csvfind -levenshtein -stop-words=&quot;the:a:of&quot; -col=1 &quot;This Red Book of West March&quot;</code></li>
+<li>E.g. <code>cat file1.csv | csvfind -inverted -levenstein -stop-words=&quot;the:a:of&quot; -col=1 &quot;This Red Book of West March&quot;</code></li>
+<li>E.g. <code>cat file1.csv | csvfind -contains -col=1 &quot;Red Book&quot;</code></li>
+</ul></li>
+<li>[x] csvjoin should have option for fuzzy match on columns (e.g. comparing titles)</li>
+</ul>
+
+</section>
+
+<footer>
+<span><h1><A href="http://caltech.edu">Caltech</a></h1></span>
+<span>&copy; 2017 <a href="https://www.library.caltech.edu/copyright">Caltech library</a></span>
+<address>1200 E California Blvd, Mail Code 1-32, Pasadena, CA 91125-3200</address> 
+<span>Phone: <a href="tel:+1-626-395-3405">(626)395-3405</a></span>
+<span><a href="mailto:[email protected]">Email Us</a></span>
+<a class="cl-hide" href="sitemap.xml">Site Map</a>
+</footer>
+</body>
+</html>
diff --git a/datatools.go b/datatools.go
@@ -35,7 +35,7 @@ import (
 )
 
 const (
-	Version = `v0.0.16-dev`
+	Version = `v0.0.16`
 
 	LicenseText = `
 %s %s

diff --git a/docs/csv2json.html b/docs/csv2json.html
@@ -60,7 +60,7 @@ <h2>EXAMPLES</h2>
 <pre><code class="language-shell">    csv2json -as-blobs -i data1.csv
 </code></pre>
 
-<p>csv2json v0.0.15</p>
+<p>csv2json v0.0.16</p>
 
 </section>
 

diff --git a/docs/csv2json.md b/docs/csv2json.md
@@ -41,4 +41,4 @@ Convert data1.csv to JSON blobs, one line per blob
 ```
 
 
-csv2json v0.0.15
+csv2json v0.0.16
diff --git a/docs/csv2mdtable.html b/docs/csv2mdtable.html
@@ -57,7 +57,7 @@ <h2>EXAMPLES</h2>
 <pre><code class="language-shell">    csv2mdtable -i data1.csv -o data1.md
 </code></pre>
 
-<p>csv2mdtable v0.0.15</p>
+<p>csv2mdtable v0.0.16</p>
 
 </section>
 

diff --git a/docs/csv2mdtable.md b/docs/csv2mdtable.md
@@ -38,4 +38,4 @@ Convert data1.csv to data1.md using options.
 ```
 
 
-csv2mdtable v0.0.15
+csv2mdtable v0.0.16
diff --git a/docs/csv2xlsx.html b/docs/csv2xlsx.html
@@ -59,7 +59,7 @@ <h2>EXAMPLE</h2>
 <p>This does the same but the contents of data.csv are piped into
 the workbook&rsquo;s sheet.</p>
 
-<p>csv2xlsx v0.0.15</p>
+<p>csv2xlsx v0.0.16</p>
 
 </section>
 

diff --git a/docs/csv2xlsx.md b/docs/csv2xlsx.md
@@ -40,4 +40,4 @@ This does the same but the contents of data.csv are piped into
 the workbook's sheet.
 
 
-csv2xlsx v0.0.15
+csv2xlsx v0.0.16
diff --git a/docs/csvcols.html b/docs/csvcols.html
@@ -76,7 +76,7 @@ <h2>EXAMPLES</h2>
 <pre><code class="language-shell">    csvcols -i 10col.csv -col 1,4,6 &gt; 3col.csv
 </code></pre>
 
-<p>csvcols v0.0.15</p>
+<p>csvcols v0.0.16</p>
 
 </section>
 

diff --git a/docs/csvcols.md b/docs/csvcols.md
@@ -60,4 +60,4 @@ Filter a 10 columns CSV file for columns 1,4,6 from file named "10col.csv"
 ```
 
 
-csvcols v0.0.15
+csvcols v0.0.16
diff --git a/docs/csvfind.html b/docs/csvfind.html
@@ -82,7 +82,7 @@ <h2>EXAMPLES</h2>
 <pre><code class="language-shell">    csvfind -i books.csv -col=2 -contains &quot;Red Book&quot;
 </code></pre>
 
-<p>csvfind v0.0.15</p>
+<p>csvfind v0.0.16</p>
 
 </section>
 

diff --git a/docs/csvfind.md b/docs/csvfind.md
@@ -64,4 +64,4 @@ You can also search for phrases in columns.
 ```
 
 
-csvfind v0.0.15
+csvfind v0.0.16
diff --git a/docs/csvjoin.html b/docs/csvjoin.html
@@ -72,7 +72,7 @@ <h2>EXAMPLES</h2>
        -output=merged-data.csv
 </code></pre>
 
-<p>csvjoin v0.0.15</p>
+<p>csvjoin v0.0.16</p>
 
 </section>
 

diff --git a/docs/csvjoin.md b/docs/csvjoin.md
@@ -51,4 +51,4 @@ merged-data.csv..
        -output=merged-data.csv
 ```
 
-csvjoin v0.0.15
+csvjoin v0.0.16
diff --git a/docs/csvrows.html b/docs/csvrows.html
@@ -77,7 +77,7 @@ <h2>EXAMPLES</h2>
 <pre><code class="language-shell">    csvrows -i 10row.csv -row 1,4,6 &gt; 3rows.csv
 </code></pre>
 
-<p>csvrows v0.0.15</p>
+<p>csvrows v0.0.16</p>
 
 </section>
 

diff --git a/docs/csvrows.md b/docs/csvrows.md
@@ -60,4 +60,4 @@ Filter a 10 row CSV file for rows 1,4,6 from file named "10row.csv"
 ```
 
 
-csvrows v0.0.15
+csvrows v0.0.16
diff --git a/docs/finddir.html b/docs/finddir.html
@@ -59,7 +59,7 @@ <h2>EXAMPLE</h2>
 
 <p>Find all subdirectories starting with &ldquo;img&rdquo;.</p>
 
-<p>finddir v0.0.15</p>
+<p>finddir v0.0.16</p>
 
 </section>
 

diff --git a/docs/finddir.md b/docs/finddir.md
@@ -39,4 +39,4 @@ finddir finds directory based on matching prefix, suffix or contained text in ba
 Find all subdirectories starting with "img". 
 
 
-finddir v0.0.15
+finddir v0.0.16
diff --git a/docs/findfile.html b/docs/findfile.html
@@ -59,7 +59,7 @@ <h2>EXAMPLE</h2>
 
 <p>Search the current directory and subdirectories for Markdown files with extension of &ldquo;.md&rdquo;.</p>
 
-<p>findfile v0.0.15</p>
+<p>findfile v0.0.16</p>
 
 </section>
 

diff --git a/docs/findfile.md b/docs/findfile.md
@@ -39,4 +39,4 @@ findfile finds files based on matching prefix, suffix or contained text in base
 Search the current directory and subdirectories for Markdown files with extension of ".md".
 
 
-findfile v0.0.15
+findfile v0.0.16
diff --git a/docs/index.html b/docs/index.html
@@ -39,6 +39,7 @@ <h1>datatools command help</h1>
 <li><a href="mergepath.html">mergepath</a></li>
 <li><a href="range.html">range</a></li>
 <li><a href="reldate.html">reldate</a></li>
+<li><a href="splitstring.html">splitstring</a></li>
 <li><a href="timefmt.html">timefmt</a></li>
 <li><a href="urlparse.html">urlparse</a></li>
 <li><a href="vcard2json.html">vcard2json</a></li>

diff --git a/docs/index.md b/docs/index.md
@@ -17,6 +17,7 @@
 + [mergepath](mergepath.html)
 + [range](range.html)
 + [reldate](reldate.html)
++ [splitstring](splitstring.html)
 + [timefmt](timefmt.html)
 + [urlparse](urlparse.html)
 + [vcard2json](vcard2json.html)

diff --git a/docs/jsoncols.html b/docs/jsoncols.html
@@ -109,7 +109,7 @@ <h2>EXAMPLES</h2>
 <pre><code class="language-csv">   &quot;Doe, Jane&quot;,[email protected],42
 </code></pre>
 
-<p>jsoncols v0.0.15</p>
+<p>jsoncols v0.0.16</p>
 
 </section>
 

diff --git a/docs/jsoncols.md b/docs/jsoncols.md
@@ -88,4 +88,4 @@ Would yield
 ```
 
 
-jsoncols v0.0.15
+jsoncols v0.0.16
diff --git a/docs/jsonjoin.html b/docs/jsonjoin.html
@@ -159,7 +159,7 @@ <h2>EXAMPLES</h2>
      &quot;bio&quot;: &quot;World renowned geophysist.&quot; }
 </code></pre>
 
-<p>jsonjoin v0.0.15</p>
+<p>jsonjoin v0.0.16</p>
 
 </section>
 

diff --git a/docs/jsonjoin.md b/docs/jsonjoin.md
@@ -149,4 +149,4 @@ would yield
 ```
 
 
-jsonjoin v0.0.15
+jsonjoin v0.0.16
diff --git a/docs/jsonmunge.html b/docs/jsonmunge.html
@@ -71,7 +71,7 @@ <h2>EXAMPLES</h2>
 <pre><code class="language-csv">    &quot;Doe, Jane&quot;
 </code></pre>
 
-<p>jsonmunge v0.0.15</p>
+<p>jsonmunge v0.0.16</p>
 
 </section>
 

diff --git a/docs/jsonmunge.md b/docs/jsonmunge.md
@@ -52,4 +52,4 @@ This would yeild
 ```
 
 
-jsonmunge v0.0.15
+jsonmunge v0.0.16
diff --git a/docs/jsonrange.html b/docs/jsonrange.html
@@ -160,7 +160,7 @@ <h2>EXAMPLES</h2>
     2
 </code></pre>
 
-<p>jsonrange v0.0.15</p>
+<p>jsonrange v0.0.16</p>
 
 </section>
 

diff --git a/docs/jsonrange.md b/docs/jsonrange.md
@@ -149,4 +149,4 @@ would yield
     2
 ```
 
-jsonrange v0.0.15
+jsonrange v0.0.16
diff --git a/docs/mergepath.html b/docs/mergepath.html
@@ -57,7 +57,7 @@ <h2>EXAMPLE</h2>
 
 <p>This would put your home bin directory at the beginning of your path.</p>
 
-<p>mergepath v0.0.15</p>
+<p>mergepath v0.0.16</p>
 
 </section>
 

diff --git a/docs/mergepath.md b/docs/mergepath.md
@@ -37,4 +37,4 @@ path and removing the resulting duplicate.
 This would put your home bin directory at the beginning of your path.
 
 
-mergepath v0.0.15
+mergepath v0.0.16
-Original file line number
+Diff line change
@@ Expand Up / @@ -35,7 +35,7 @@ import ( @@
     )
     const (
-    	Version = `v0.0.16-dev`
+    	Version = `v0.0.16`
     	LicenseText = `
     %s %s
@@ Expand Down @@
Original file line number	Diff line number	Diff line change
Expand Up		@@ -41,4 +41,4 @@ Convert data1.csv to JSON blobs, one line per blob
		```


		csv2json v0.0.15
		csv2json v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -38,4 +38,4 @@ Convert data1.csv to data1.md using options.
		```


		csv2mdtable v0.0.15
		csv2mdtable v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -40,4 +40,4 @@ This does the same but the contents of data.csv are piped into
		the workbook's sheet.


		csv2xlsx v0.0.15
		csv2xlsx v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -60,4 +60,4 @@ Filter a 10 columns CSV file for columns 1,4,6 from file named "10col.csv"
		```


		csvcols v0.0.15
		csvcols v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -64,4 +64,4 @@ You can also search for phrases in columns.
		```


		csvfind v0.0.15
		csvfind v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -60,4 +60,4 @@ Filter a 10 row CSV file for rows 1,4,6 from file named "10row.csv"
		```


		csvrows v0.0.15
		csvrows v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -39,4 +39,4 @@ finddir finds directory based on matching prefix, suffix or contained text in ba
		Find all subdirectories starting with "img".


		finddir v0.0.15
		finddir v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -39,4 +39,4 @@ findfile finds files based on matching prefix, suffix or contained text in base
		Search the current directory and subdirectories for Markdown files with extension of ".md".


		findfile v0.0.15
		findfile v0.0.16
Original file line number	Diff line number	Diff line change
Expand Up		@@ -88,4 +88,4 @@ Would yield
		```


		jsoncols v0.0.15
		jsoncols v0.0.16