Skip to content

Commit

Permalink
use as go module, win support (#21)
Browse files Browse the repository at this point in the history
  • Loading branch information
AskAlexSharov authored Jun 19, 2021
1 parent b811f2d commit eea7f5a
Show file tree
Hide file tree
Showing 42 changed files with 36,767 additions and 1,476 deletions.
28 changes: 19 additions & 9 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,12 @@ jobs:
strategy:
matrix:
os: [ ubuntu-16.04, ubuntu-18.04, ubuntu-20.04, macos-10.15 ] # list of os: https://github.com/actions/virtual-environments
# os: [ubuntu-20.04, macos-10.15, windows-latest] # list of os: https://github.com/actions/virtual-environments
go: [ '1.16', '1.13' ]
go: [ '1.16', '1.15' ]
runs-on: ${{ matrix.os }}

steps:
- uses: actions/checkout@v2

- name: Add Msys64 to PATH # see https://github.com/actions/virtual-environments/issues/1613 and https://github.com/actions/virtual-environments/pull/1648
if: matrix.os == 'windows-latest'
run: echo "::add-path::/c/msys64/mingw64/bin:/c/msys64/usr/bin"
shell: bash

- uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
Expand All @@ -27,8 +21,24 @@ jobs:
path: ~/go/pkg/mod
key: go-${{ matrix.os }}-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}

- name: Test
run: cc --version && make test
- run: go test -v ./...

- name: Race
run: make race

win:
strategy:
matrix:
os: [ windows-2019 ] # list of os: https://github.com/actions/virtual-environments
go: [ '1.16' ]
runs-on: ${{ matrix.os }}

steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}

- run: choco upgrade mingw cmake -y --no-progress
- run: go test -v ./...
4 changes: 1 addition & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# c repository for updating after each release
/lmdb/lmdb
/lmdb/openldap
.idea

/bin/
/build
Expand Down
4 changes: 0 additions & 4 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,13 @@ linters:
- errcheck
- goconst
- gofmt
- golint
- interfacer
- govet
- structcheck
- stylecheck
- staticcheck
# - goerr113
- unconvert
- unparam
- scopelint
- nakedret
- prealloc
- gosimple
Expand All @@ -31,7 +28,6 @@ linters:
- depguard
- typecheck
- misspell
- maligned

linters-settings:
gofmt:
Expand Down
14 changes: 5 additions & 9 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ GOLDFLAGS="-X main.branch $(BRANCH) -X main.commit $(COMMIT)"
deps: lintci-deps
go get -d ./...

all: deps check mdbx-build
all: deps check

test: mdbx-build
test:
go test ./mdbx ./exp/mdbxpool

race: mdbx-build
race:
go test -race ./mdbx ./exp/mdbxpool

lint: mdbx-build
lint:
./build/bin/golangci-lint run --new-from-rev=$(MASTER_COMMIT) ./...

lintci-deps:
rm -f ./build/bin/golangci-lint
curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b ./build/bin v1.31.0
curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b ./build/bin v1.41.0

check:
which goimports > /dev/null
Expand All @@ -32,7 +32,3 @@ check:

clean:
cd mdbx/dist/ && make clean

mdbx-build:
echo "Building mdbx"
cd mdbx/dist/ && make clean && make config.h && CFLAGS_EXTRA="-Wno-deprecated-declarations" make mdbx-static.o
170 changes: 65 additions & 105 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,23 @@
# mdbx-go
# mdbx-go

Go bindings to the libmdbx: https://github.com/erthink/libmdbx
Go bindings to the libmdbx: https://erthink.github.io/libmdbx/

Most of articles in internet about LMDB are applicable to MDBX. But mdbx has more features.

For deeper DB understanding please read through [mdbx.h](https://github.com/erthink/libmdbx/blob/master/mdbx.h)

## Min Requirements

C language Compilers compatible with GCC or CLANG (mingw 10 on windows)
Golang: 1.15

## Packages

Functionality is logically divided into several packages. Applications will
usually need to import **mdbx** but may import other packages on an as needed
basis.
Functionality is logically divided into several packages. Applications will usually need to import **mdbx** but may
import other packages on an as needed basis.

Packages in the `exp/` directory are not stable and may change without warning.
That said, they are generally usable if application dependencies are managed
and pinned by tag/commit.
Packages in the `exp/` directory are not stable and may change without warning. That said, they are generally usable if
application dependencies are managed and pinned by tag/commit.

Developers concerned with package stability should consult the documentation.

Expand All @@ -24,59 +31,34 @@ Core bindings allowing low-level access to MDBX.

#### exp/mdbxpool [![GoDoc](https://godoc.org/github.com/torquem-ch/mdbx-go/mdbx/exp/mdbxpool?status.svg)](https://godoc.org/github.com/torquem-ch/mdbx-go/mdbx/exp/mdbxpool) [![experimental](https://img.shields.io/badge/stability-experimental-red.svg)](#user-content-versioning-and-stability)


```go
import "github.com/torquem-ch/mdbx-go/exp/mdbxpool"
```

A utility package which facilitates reuse of mdbx.Txn objects using a
sync.Pool. Naively storing mdbx.Txn objects in sync.Pool can be troublesome.
And the mdbxpool.TxnPool type has been defined as a complete pooling solution
and as reference for applications attempting to write their own pooling
implementation.

The mdbxpool package is relatively new. But it has a lot of potential utility.
And once the mdbxpool API has been ironed out, and the implementation hardened
through use by real applications it can be integrated directly into the mdbx
package for more transparent integration. Please test this package and provide
feedback to speed this process up.
A utility package which facilitates reuse of mdbx.Txn objects using a sync.Pool. Naively storing mdbx.Txn objects in
sync.Pool can be troublesome. And the mdbxpool.TxnPool type has been defined as a complete pooling solution and as
reference for applications attempting to write their own pooling implementation.

The mdbxpool package is relatively new. But it has a lot of potential utility. And once the mdbxpool API has been ironed
out, and the implementation hardened through use by real applications it can be integrated directly into the mdbx
package for more transparent integration. Please test this package and provide feedback to speed this process up.

## Key Features

### Idiomatic API

API inspired by [BoltDB](https://github.com/boltdb/bolt) with automatic
commit/rollback of transactions. The goal of mdbx-go is to provide idiomatic
database interactions without compromising the flexibility of the C API.

**NOTE:** While the mdbx package tries hard to make MDBX as easy to use as
possible there are compromises, gotchas, and caveats that application
developers must be aware of when relying on MDBX to store their data. All
users are encouraged to fully read the
[documentation](https://godoc.org/github.com/torquem-ch/mdbx-go/mdbx) so they are
aware of these caveats.

### API coverage

The mdbx-go project aims for complete coverage of the MDBX C API (within
reason). Some notable features and optimizations that are supported:

- Idiomatic subtransactions ("sub-updates") that allow the batching of updates.

- Batch IO on databases utilizing the `MDB_DUPSORT` and `MDB_DUPFIXED` flags.
API inspired by [BoltDB](https://github.com/boltdb/bolt) with automatic commit/rollback of transactions. The goal of
mdbx-go is to provide idiomatic database interactions without compromising the flexibility of the C API.

- Reserved writes than can save in memory copies converting/buffering into
`[]byte`.
**NOTE:** While the mdbx package tries hard to make MDBX as easy to use as possible there are compromises, gotchas, and
caveats that application developers must be aware of when relying on MDBX to store their data. All users are encouraged
to fully read the [documentation](https://erthink.github.io/libmdbx/) so they are aware of these caveats. And even
better if read through [mdbx.h](https://github.com/erthink/libmdbx/blob/master/mdbx.h)

For tracking purposes a list of unsupported features is kept in an
[issue](https://github.com/torquem-ch/mdbx-go/issues/1).
### High Performance notices

### Zero-copy reads

Applications with high performance requirements can opt-in to fast, zero-copy
reads at the cost of runtime safety. Zero-copy behavior is specified at the
transaction level to reduce instrumentation overhead.
Applications with high performance requirements can opt-in to fast, zero-copy reads at the cost of runtime safety.
Zero-copy behavior is specified at the transaction level to reduce instrumentation overhead.

```
err := mdbx.View(func(txn *mdbx.Txn) error {
Expand All @@ -89,91 +71,69 @@ err := mdbx.View(func(txn *mdbx.Txn) error {
})
```

## MDBX compared to BoltDB

BoltDB is a quality database with a design similar to MDBX. Both store
key-value data in a file and provide ACID transactions. So there are often
questions of why to use one database or the other.
Use NoReadahead if Data > RAM

### Advantages of BoltDB

- Nested databases allow for hierarchical data organization.

- Far more databases can be accessed concurrently.

- Operating systems that do not support sparse files do not use up excessive
space due to a large pre-allocation of file space.
- No `Bucket` object - means less allocations and higher performance

- Operating systems that do not support sparse files do not use up excessive space due to a large pre-allocation of file
space.

- As a pure Go package bolt can be easily cross-compiled using the `go`
toolchain and `GOOS`/`GOARCH` variables.

- Its simpler design and implementation in pure Go mean it is free of many
caveats and gotchas which are present using the MDBX package. For more
information about caveats with the MDBX package, consult its
[documentation](https://godoc.org/github.com/torquem-ch/mdbx-go/mdbx).
- Its simpler design and implementation in pure Go mean it is free of many caveats and gotchas which are present using
the MDBX package. For more information about caveats with the MDBX package, consult its
[documentation](https://erthink.github.io/libmdbx/) so they are aware of these caveats. And even better if read
through [mdbx.h](https://github.com/erthink/libmdbx/blob/master/mdbx.h).

### Advantages of LMDB and MDBX
### Advantages of MDBX

- Keys can contain multiple values using the DupSort flag.

- Updates can have sub-updates for atomic batching of changes.

- Databases typically remain open for the application lifetime. This limits
the number of concurrently accessible databases. But, this minimizes the
overhead of database accesses and typically produces cleaner code than
an equivalent BoltDB implementation.

- Significantly faster than BoltDB. The raw speed of MDBX easily surpasses
BoltDB. Additionally, MDBX provides optimizations ranging from safe,
feature-specific optimizations to generally unsafe, extremely situational
ones. Applications are free to enable any optimizations that fit their data,
access, and reliability models.

- MDBX allows multiple applications to access a database simultaneously.
Updates from concurrent processes are synchronized using a database lock
file.

- As a C library, applications in any language can interact with MDBX
databases. Mission critical Go applications can use a database while Python
scripts perform analysis on the side.

## Build
- Databases typically remain open for the application lifetime. This limits the number of concurrently accessible
databases. But, this minimizes the overhead of database accesses and typically produces cleaner code than an
equivalent BoltDB implementation.

There is no dependency on shared libraries. But it's impossible to use 'go get' for now. Only way is to copy sources of this package to your project, and call `make mdbx-build` manually. See: https://github.com/torquem-ch/mdbx-go/issues/5
- Significantly faster than BoltDB. The raw speed of MDBX easily surpasses BoltDB. Additionally, MDBX provides
optimizations ranging from safe, feature-specific optimizations to generally unsafe, extremely situational ones.
Applications are free to enable any optimizations that fit their data, access, and reliability models.

On FreeBSD 10, you must explicitly set `CC` (otherwise it will fail with a
cryptic error), for example:
- MDBX allows multiple applications to access a database simultaneously. Updates from concurrent processes are
synchronized using a database lock file.

CC=clang go test -v ./...
- As a C library, applications in any language can interact with MDBX databases. Mission critical Go applications can
use a database while Python scripts perform analysis on the side.

## Documentation

### Go doc

The `go doc` documentation available on
[godoc.org](https://godoc.org/github.com/torquem-ch/mdbx-go) is the primary source
of developer documentation for mdbx-go. It provides an overview of the API
with a lot of usage examples. Where necessary the documentation points out
differences between the semantics of methods and their C counterparts.
## Build

### LMDB
On FreeBSD 10, you must explicitly set `CC` (otherwise it will fail with a cryptic error), for example:

The LMDB [homepage](http://symas.com/mdb/)
CC=clang go test -v ./...

### MDBX
## Documentation

The MDBX [homepage](https://github.com/erthink/libmdbx)
- Examples see in *_test.go files of this repo
- [The MDBX](https://erthink.github.io/libmdbx/) And even better if read
through [mdbx.h](https://github.com/erthink/libmdbx/blob/master/mdbx.h).
- [godoc.org](https://godoc.org/github.com/torquem-ch/mdbx-go)
- [The LMDB](http://symas.com/mdb/)

### Versioning and Stability

The mdbx-go project makes regular releases with IDs `X.Y.Z`. All packages
outside of the `exp/` directory are considered stable and adhere to the
guidelines of [semantic versioning](http://semver.org/).
The mdbx-go project makes regular releases with IDs `X.Y.Z`. All packages outside of the `exp/` directory are considered
stable and adhere to the guidelines of [semantic versioning](http://semver.org/).

Experimental packages (those packages in `exp/`) are not required to adhere to
semantic versioning. However packages specifically declared to merely be
Experimental packages (those packages in `exp/`) are not required to adhere to semantic versioning. However packages
specifically declared to merely be
"unstable" can be relied on more for long term use with less concern.

The API of an unstable package may change in subtle ways between minor release
versions. But deprecations will be indicated at least one release in advance
and all functionality will remain available through some method.
The API of an unstable package may change in subtle ways between minor release versions. But deprecations will be
indicated at least one release in advance and all functionality will remain available through some method.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
module github.com/torquem-ch/mdbx-go

go 1.13
go 1.15
37 changes: 0 additions & 37 deletions internal/mdbxcmd/cmutil.go

This file was deleted.

1 change: 1 addition & 0 deletions mdbx/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
*.dylib
*.o
config.h
config2.h
config_darwin.h
config_linux.h
config_win.h
Expand Down
Loading

0 comments on commit eea7f5a

Please sign in to comment.