Skip to content

Commit

Permalink
Update README
Browse files Browse the repository at this point in the history
- Add licence and pre-commit script
- Update go modules for examples
  • Loading branch information
shivamMg committed Dec 19, 2018
1 parent 4f235fe commit e6841a1
Show file tree
Hide file tree
Showing 10 changed files with 87 additions and 22 deletions.
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2018 Shivam Mamgain

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
60 changes: 42 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# rd
# rd [![godoc](https://godoc.org/github.com/shivammg/rd?status.svg)](https://godoc.org/github.com/shivamMg/rd) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

`rd` is a small library to build hand-written recursive descent parsers. Besides exposing convenient methods to parse tokens it features automatic parse tree generation and flow tracing for debugging.

Recursive descent parsers can imitate their grammar quite well. e.g. for the grammar:
Recursive descent parsers can imitate their grammar quite well. For instance for the following grammar:

```
A → aB
B → b
```

where `A` and `B` are non-terminals, and `a` and `b` are terminals, a recursive descent parser might look like:
(where `A` and `B` are non-terminals, and `a` and `b` are terminals), a recursive descent parser might look like:

```go
func A() bool {
Expand All @@ -32,35 +32,36 @@ func B() bool {
Parser for the same grammar written using `rd`:

```go
import "github.com/shivamMg/rd"

func A(b *rd.Builder) (ok bool) {
b.Enter("A")
defer b.Exit(&ok)
defer b.Exit(&ok)

return b.Match("a") && B(b)
return b.Match("a") && B(b)
}

func B(b *rd.Builder) (ok bool) {
defer b.Enter("B").Exit(&ok)
defer b.Enter("B").Exit(&ok)

return b.Match("b")
return b.Match("b")
}
```

`Match` method conveniently resets original state in case of failure terminal matches. `Enter` and `Exit` methods serve the purpose of marking entry and exit respectively from the non-terminal functions. Argument to `Enter`, e.g. `"A"`, is what will show up in the parse tree. The exit result (`&ok`) determines if the generated subtree for the non-terminal must be added to the parse tree. This subtree is created considering successful terminal matches and non-terminals calls done inside the non-terminal.
A builder object keeps track of the current token and exposes convenient methods to write a parser. `Match`, for instance, resets the original state in case of an unsuccessful match so there's no need for a manual `pushback`. As non-terminal functions are called, terminal matches, and calls to other non-terminal functions are done. A parse tree is maintained in the builder using these matches and calls:

In case of a successful match the terminal is added to parse tree under the current non-terminal (the one in which `Match` was called). Same goes in case of a non-terminal function call: the non-terminal, if it exits successfully, is added to parse tree under the current non-terminal. You can imagine this process being repeated recursively.

A debug tree is also generated that contains all non-terminal calls and terminal matches - unlike the parse tree that contains only successful ones. Debug tree is helpful if you want to debug a parsing failure.
Argument to `Enter` is what is added to parse tree as a symbol for the non-terminal. Argument to `Exit` determines if function call was successful or not.

Both parse trees and debug trees satisfy `fmt.Stringer` and can be pretty printed.
`ParseTree` method returns the parse tree which is, well, a tree data structure. It can be pretty-printed.

```go
tokens := []rd.Token{"a", "b"}
b := rd.NewBuilder(tokens)
if ok := A(b); ok {
fmt.Println(b.ParseTree())
} else {
fmt.Println(b.Err())
fmt.Print(b.ParseTree())
}
fmt.Println(b.DebugTree())
```

The above snippet will print:
Expand All @@ -70,14 +71,31 @@ A
├─ a
└─ B
└─ b
```

A debug tree is also maintained which, unlike the parse tree, contains all matches and calls (not just the successful ones). It's helpful if you want to debug a parsing failure.

```go
fmt.Print(b.DebugTree())
```

The above snippet will print:

```
A(true)
├─ a = a
└─ B(true)
└─ b = b
```

For `tokens := []rd.Token{"a", "c"}` the snippet will print:
Parsing errors can be retrieved using `Err` method. For `tokens := []rd.Token{"a", "c"}` the following statements:

```go
fmt.Println(b.Err())
fmt.Print(b.DebugTree())
```

will print:

```
parsing error
Expand All @@ -87,8 +105,6 @@ A(false)
└─ c ≠ b
```

In the debug tree you can notice both successful and unsuccessful terminal matches, and non-terminal exit results.


## Examples

Expand All @@ -100,7 +116,7 @@ arithmetic -expr='3.14*4*(6/3)' # hopefully $GOPATH/bin is in $PATH
arithmetic -expr='3.14*4*(6/3)' -backtrackingparser
```

Parser and grammar can be found inside `examples/arithmetic/parser`. There's another parser written for a different grammar that also parses arithmetic expressions. This parser can be found inside `examples/arithmetic/backtrackingparser`. It uses backtracking - notice the use of `b.Backtrack()`. This example uses [chroma](https://github.com/alecthomas/chroma) for lexing.
Parser and grammar for it can be found inside `examples/arithmetic/parser`. There's another parser written for a different grammar that also parses arithmetic expressions. This parser can be found inside `examples/arithmetic/backtrackingparser`. It uses backtracking - notice the use of `b.Backtrack()`. This example uses [chroma](https://github.com/alecthomas/chroma) for lexing.


### PL/0 programming language parser
Expand All @@ -124,3 +140,11 @@ domainname www.google.co.uk

Grammar has been taken from [www.ietf.org/rfc/rfc1035.txt](https://www.ietf.org/rfc/rfc1035.txt). Its lexer is hand-written.

## Licence

MIT

## Contribute

Contribute through bug fixes, improvements, new examples, etc. Lucky PR submitters get to walk home with a brand new CRT and an Audi 1987.

2 changes: 1 addition & 1 deletion builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ func (b *Builder) Skip() {
// under this non-terminal.
//
// Enter should be called right after entering the non-terminal function.
func (b *Builder) Enter(nonTerm Token) *Builder {
func (b *Builder) Enter(nonTerm interface{}) *Builder {
b.stack.push(ele{
index: b.current,
nonTerm: NewTree(nonTerm),
Expand Down
2 changes: 1 addition & 1 deletion examples/arithmetic/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ module github.com/shivamMg/rd/examples/arithmetic
require (
github.com/alecthomas/chroma v0.6.0
github.com/dlclark/regexp2 v1.1.6 // indirect
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5
github.com/shivamMg/rd v0.0.0
)
2 changes: 2 additions & 0 deletions examples/arithmetic/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@ github.com/shivamMg/ppds v0.0.0-20180628070107-c32714a96b1e h1:6TC4+mcfjw9olnx6z
github.com/shivamMg/ppds v0.0.0-20180628070107-c32714a96b1e/go.mod h1:hb39VqUO6qfkb9zBBQPTIV1vWBtI7yQsG0wr3pN78fM=
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5 h1:aWs3t5hco+TE+KVYsmSHdHLK4Apu2FcbAtA+vu6+HaE=
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5/go.mod h1:JneoUABwp5pIIg7fOhS1sWNSSPjrM7tddQTe1Mf5c78=
github.com/shivamMg/rd v0.0.0 h1:RpdJI/Aj/2A5fn3ub/Z7JNIXri+lutYFI+XKUS697OQ=
github.com/shivamMg/rd v0.0.0/go.mod h1:JneoUABwp5pIIg7fOhS1sWNSSPjrM7tddQTe1Mf5c78=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
2 changes: 1 addition & 1 deletion examples/domainname/go.mod
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
module github.com/shivamMg/rd/examples/domainname

require github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5 // indirect
require github.com/shivamMg/rd v0.0.0 // indirect
3 changes: 3 additions & 0 deletions examples/domainname/go.sum
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/shivamMg/ppds v0.0.0-20180628070107-c32714a96b1e h1:6TC4+mcfjw9olnx6zfSsq1GA9/MnCuF84vW9MK1PgYs=
github.com/shivamMg/ppds v0.0.0-20180628070107-c32714a96b1e/go.mod h1:hb39VqUO6qfkb9zBBQPTIV1vWBtI7yQsG0wr3pN78fM=
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5 h1:aWs3t5hco+TE+KVYsmSHdHLK4Apu2FcbAtA+vu6+HaE=
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5/go.mod h1:JneoUABwp5pIIg7fOhS1sWNSSPjrM7tddQTe1Mf5c78=
github.com/shivamMg/rd v0.0.0 h1:RpdJI/Aj/2A5fn3ub/Z7JNIXri+lutYFI+XKUS697OQ=
github.com/shivamMg/rd v0.0.0/go.mod h1:JneoUABwp5pIIg7fOhS1sWNSSPjrM7tddQTe1Mf5c78=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
2 changes: 1 addition & 1 deletion examples/pl0/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ module github.com/shivamMg/rd/examples/pl0
require (
github.com/alecthomas/chroma v0.6.0
github.com/dlclark/regexp2 v1.1.6 // indirect
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5
github.com/shivamMg/rd v0.0.0
)
2 changes: 2 additions & 0 deletions examples/pl0/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@ github.com/shivamMg/ppds v0.0.0-20180628070107-c32714a96b1e h1:6TC4+mcfjw9olnx6z
github.com/shivamMg/ppds v0.0.0-20180628070107-c32714a96b1e/go.mod h1:hb39VqUO6qfkb9zBBQPTIV1vWBtI7yQsG0wr3pN78fM=
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5 h1:aWs3t5hco+TE+KVYsmSHdHLK4Apu2FcbAtA+vu6+HaE=
github.com/shivamMg/rd v0.0.0-20181206194526-a7fe8c9f63b5/go.mod h1:JneoUABwp5pIIg7fOhS1sWNSSPjrM7tddQTe1Mf5c78=
github.com/shivamMg/rd v0.0.0 h1:RpdJI/Aj/2A5fn3ub/Z7JNIXri+lutYFI+XKUS697OQ=
github.com/shivamMg/rd v0.0.0/go.mod h1:JneoUABwp5pIIg7fOhS1sWNSSPjrM7tddQTe1Mf5c78=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
13 changes: 13 additions & 0 deletions pre-commit.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/sh

GOFMT=$(gofmt -l .)
if [ -n "${GOFMT}" ]; then
printf >&2 'gofmt failed for:\n%s\n' "${GOFMT}"
exit 1
fi

GOTEST=$(go test ./...)
if [ $? -ne 0 ]; then
printf >&2 'go test failed:\n%s\n' "${GOTEST}"
exit 1
fi

0 comments on commit e6841a1

Please sign in to comment.