-
Notifications
You must be signed in to change notification settings - Fork 81
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
82 additions
and
138 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,17 +9,9 @@ Deephaven includes an intuitive user experience and visualization tools. | |
| It can ingest data from a variety of sources, apply computation and analysis algorithms | ||
| to that data, and build rich queries, dashboards, and representations with the results. | ||
|
|
||
| Deephaven Community Core is an open version of [Deephaven Enterprise](https://deephaven.io), | ||
| Deephaven Community Core is the open version of [Deephaven Enterprise](https://deephaven.io), | ||
| which functions as the data backbone for prominent hedge funds, banks, and financial exchanges. | ||
|
|
||
| [](https://gitter.im/deephaven/deephaven?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
|
|
||
| ## Supported Languages | ||
|
|
||
| | Language | Server Application | Client Application | | ||
|
|
@@ -30,200 +22,148 @@ which functions as the data backbone for prominent hedge funds, banks, and finan | |
| | JavaScript | No | Yes | | ||
| | Go | No | Yes | | ||
| | R | No | Yes | | ||
| | gRPC | - | Yes | | ||
|
|
||
| ## Run Deephaven | ||
|
|
||
| There are both server- and client-side applications available, as seen in the table above. | ||
|
|
||
| ### Server-side | ||
| ## Installation | ||
|
|
||
| Deephaven's server-side APIs allow you to connect directly to the Deephaven server and execute commands, write queries, and much more. | ||
| Deephaven Community Core can be used from both server- and client-side APIs. Server-side APIs allow users to connect directly to the Deephaven server and execute commands, write queries, and much more. Client-side APIs allow users to connect to a Deephaven server through a client application. | ||
|
|
||
| Most users will want to run Deephaven from pre-built images. It's the easiest way to configure, deploy, and use Deephaven. For detailed instructions, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart). | ||
| See the following documentation links for installation instructions and more: | ||
|
|
||
| Some Python users will want to use Deephaven entirely from Python. For instructions on starting and using Deephaven from Python, see [Quick start for pip-installed Deephaven](https://deephaven.io/core/docs/tutorials/quickstart-pip/). | ||
| - Python | ||
| - [Run from Docker](https://deephaven.io/core/docs/tutorials/quickstart/) | ||
| - [pip-installed](https://deephaven.io/core/docs/tutorials/quickstart/) | ||
| - [Groovy] | ||
| - [Run from Docker](https://deephaven.io/core/groovy/docs/tutorials/quickstart/) | ||
| - [Python client](https://pypi.org/project/pydeephaven/) | ||
| - [Java client](https://deephaven.io/core/docs/how-to-guides/java-client/) | ||
| - [JS client](https://deephaven.io/core/docs/reference/js-api/documentation/) | ||
| - [Go client](https://pkg.go.dev/github.com/deephaven/deephaven-core/go) | ||
| - [R client](https://github.com/deephaven/deephaven-core/blob/main/R/rdeephaven/README.md) | ||
|
|
||
| Developers interested in tinkering with and modifying source code should build from source. For detailed instructions on how to do this, see [Build and launch Deephaven](https://deephaven.io/core/docs/how-to-guides/launch-build). | ||
| Deephaven's client APIs use [gRPC](https://grpc.io/), [protobuf](https://github.com/deephaven/deephaven-core/tree/main/proto/proto-backplane-grpc/src/main/proto/deephaven/proto), [Apache Arrow Flight](https://arrow.apache.org/docs/format/Flight.html), and [Barrage](https://github.com/deephaven/barrage) to handle ticking data. Users who wish to build their own client APIs can use these tools to do so. | ||
|
|
||
| If you are not sure which of the two is right for you, use the pre-built images. | ||
| ## Build from source | ||
|
|
||
| ### Client-side | ||
| Users who wish to modify or contribute to this project should [build from source](https://deephaven.io/core/docs/how-to-guides/launch-build/) to run either the Python or Groovy server-side API. | ||
|
|
||
| Deephaven's client APIs allow you to connect to a Deephaven server and execute code. Documentation links are below: | ||
| Start by cloning this repository, and `cd` into the cloned repo. | ||
|
|
||
| - [Python](https://deephaven.io/core/client-api/python/) | ||
| - [Java/Groovy](https://deephaven.io/core/javadoc/) | ||
| - [C++](https://deephaven.io/core/client-api/cpp/) | ||
| - [JavaScript](https://deephaven.io/core/docs/reference/js-api/documentation/) | ||
| - [Go](https://pkg.go.dev/github.com/deephaven/deephaven-core/go) | ||
| - Documentation for R can be found with the `?` operator in R. | ||
| ```bash | ||
| git clone https://github.com/deephaven/deephaven-core.git | ||
| cd deephaven-core | ||
| ``` | ||
|
|
||
| ### Docker Dependencies | ||
| ### Required dependencies | ||
|
|
||
| Running Deephaven from pre-built Docker images requires a few software packages. | ||
| Building and running Deephaven requires a few software packages. | ||
|
|
||
| | Package | Version | OS | | ||
| | -------------- | ----------------------------- | ------------ | | ||
| | git | ^2.25.0 | All | | ||
| | java | >=11, <20 | All | | ||
| | docker | ^20.10.8 | All | | ||
| | docker-compose | ^1.29.0 | All | | ||
| | Windows | 10 (OS build 20262 or higher) | Only Windows | | ||
| | WSL | 2 | Only Windows | | ||
|
|
||
| You can check if these packages are installed and functioning by running: | ||
| ``` | ||
|
|
||
| ```bash | ||
| git version | ||
| java -version | ||
| docker version | ||
| docker-compose version | ||
| docker run hello-world | ||
| ``` | ||
|
|
||
| > :warning: **On Windows, all commands must be run inside a WSL 2 terminal.** | ||
| :::note | ||
|
|
||
| If any dependencies are missing or unsupported versions are installed, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart#prerequisites) for installation instructions. | ||
| Internally, the Java build process will use [Gradle Auto Provisioning](https://docs.gradle.org/current/userguide/toolchains.html#sec:provisioning) | ||
| to download and use the appropriate Java version for building and testing. | ||
|
|
||
| On Windows, all commands must be run inside a WSL 2 terminal. | ||
|
|
||
| ### Create deployment | ||
| ::: | ||
|
|
||
| A directory must be created to store files and mount points for your deployment. Here, we are using the `deephaven-deployment` directory. | ||
| ### Python | ||
|
|
||
| You will need to `cd` into the deployment directory to launch or interact with the deployment. | ||
| Set up a virtual environment. | ||
|
|
||
| ```bash | ||
| mkdir deephaven-deployment | ||
| cd deephaven-deployment | ||
| python -m venv /tmp/my-dh-venv | ||
| source /tmp/my-dh-venv/bin/activate | ||
| ``` | ||
|
|
||
| > :warning: **Commands in the following sections for interacting with a deployment must be run from the deployment directory.** | ||
| ### Launch options | ||
|
|
||
| Deephaven offers a variety of pre-made Docker Compose YAML files for different deployments. The following commands will launch Deephaven. Replace `<URL>` with that of the deployment from the table below that best suits your needs. | ||
| Then, build and install the wheel. | ||
|
|
||
| ```bash | ||
| curl <URL> -O | ||
| docker compose pull | ||
| docker compose up -d | ||
| ./gradlew py-server:assemble | ||
| pip install "py/server/build/wheel/deephaven_core-<version>-py3-non-any.whl[autocomplete] | ||
| ``` | ||
| | Language | Example data | Additional packages | URL | | ||
| | Python | No | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/base/docker-compose.yml | | ||
| | Python | No | SciKit-Learn | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/SciKit-Learn/docker-compose.yml | | ||
| | Python | No | PyTorch | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml | | ||
| | Python | No | TensorFlow | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/TensorFlow/docker-compose.yml | | ||
| | Python | No | NLTK | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml | | ||
| | Python | Yes | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml | | ||
| | Python | Yes | SciKit-Learn | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml | | ||
| | Python | Yes | PyTorch | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml | | ||
| | Python | Yes | TensorFlow | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml | | ||
| | Python | Yes | NLTK | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml | | ||
| | Groovy | No | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml | | ||
| | Groovy | Yes | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml | | ||
|
|
||
| Where: | ||
| ### Monitor logs | ||
| - `<version>` is replaced by a deephaven-core [release](https://github.com/deephaven/deephaven-core/releases). | ||
| - Users who wish to use Deephaven without autocomplete should remove `[autocomplete]` from the `pip install` command. | ||
| The `-d` option to `docker-compose` causes the containers to run in the background, in detached mode. This option allows you to use your shell after Docker launches the containers. | ||
|
|
||
| Since the container is running detached, you will not see any logs. However, you can follow the logs by running: | ||
| Lastly, build. | ||
| ```bash | ||
| docker-compose logs -f | ||
| ./gradlew server-jetty-app:run | ||
| ``` | ||
| Use CTRL+C to stop monitoring the logs and return to a prompt. | ||
|
|
||
| ### Authentication | ||
|
|
||
| Deephaven, by default, comes stock with pre-shared key to authenticate any users trying to access it. If no key is set, a randomly generated key is printed to the Docker logs. The key can be found by monitoring the logs. It will look like this: | ||
| ### Groovy/Java | ||
| ``` | ||
| ================================================================================ | ||
| Superuser access through pre-shared key is enabled - use 1c3opvgzl5scm to connect | ||
| Connect automatically to Web UI with http://localhost:10000/?psk=1c3opvgzl5scm | ||
| ================================================================================ | ||
| ``` | ||
|
|
||
| You can also `grep` for the key in the logs: | ||
| This single command will build and run the Deephaven Groovy server-side API from source. | ||
| ```bash | ||
| docker compose logs -f | grep "access through pre-shared key" | ||
| ./gradlew server-jetty-app:run | ||
| ``` | ||
| For more information on setting your own pre-shared key, see [How to configure and use pre-shared key authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-psk/). | ||
|
|
||
| For information on enabling anonymous authentication, see [How to enable anonymous authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-anon/). | ||
|
|
||
| ### Shutdown | ||
|
|
||
| The deployment can be brought down by running: | ||
|
|
||
| ```bash | ||
| docker-compose down | ||
| ``` | ||
|
|
||
| ### Manage example data | ||
|
|
||
| [Deephaven's examples repository](https://github.com/deephaven/examples) contains data sets that are useful when learning | ||
| to use Deephaven. These data sets are used extensively in Deephaven's documentation and are needed to run some examples. [Deephaven's examples repository](https://github.com/deephaven/examples) contains documentation on the available data sets and how to manage them. | ||
|
|
||
| If you have chosen a deployment with example data, the example data sets will be downloaded. Production deployments containing your own data will not need the example data sets. | ||
|
|
||
| ### Authentication | ||
| To upgrade a deployment to the latest example data, run: | ||
| Deephaven, by default, uses pre-shared key authentication to authenticate against unauthorized access. Users will be prompted for a key when connecting to an instance of the server. Unless otherwise specified, a new randomly generated key will be used each time the server is started. The key is printed to the Docker logs. To search the logs for the key, run: | ||
| ```bash | ||
| docker-compose run examples download | ||
| docker compose logs -f | grep "access through pre-shared key" | ||
| ``` | ||
| To see what other example data management commands are available, run: | ||
| To set the key, add `-Ppsk` to the build command: | ||
| ```bash | ||
| docker-compose run examples | ||
| ./gradlew server-jetty-app:run -Ppsk=My-Password | ||
| ``` | ||
| If your deployment does not have example data, these commands will fail with `ERROR: No such service`. | ||
|
|
||
| ### Build from source | ||
|
|
||
| If you are interested in tinkering and modifying source code for your own Deephaven deployment, or contributing to the project, see this [README](https://github.com/deephaven/deephaven-core/blob/main/server/jetty-app/README.md) for build instructions. There are several [required dependencies](https://deephaven.io/core/docs/how-to-guides/launch-build/#required-dependencies) to build and launch from source code. | ||
| ### Connect to the server | ||
| Additionally, see our [code of conduct](https://github.com/deephaven/deephaven-core/blob/main/CODE_OF_CONDUCT.md) and [contributing guide](https://github.com/deephaven/deephaven-core/blob/main/CONTRIBUTING.md). | ||
|
|
||
| ## Run Deephaven IDE | ||
|
|
||
| Once Deephaven is running, you can launch a Deephaven IDE in your web browser. Deephaven IDE allows you to interactively analyze data. | ||
|
|
||
| - If Deephaven is running locally, navigate to [http://localhost:10000/ide/](http://localhost:10000/ide/). | ||
| - If Deephaven is running remotely, navigate to `http://<hostname>:10000/ide/`, where `<hostname>` is the address of the machine Deephaven is running on. | ||
|
|
||
|  | ||
| Deephaven is run from a web browser, and can be connected to via `http://localhost:10000/ide`. | ||
| ## First query | ||
| From the Deephaven IDE, you can perform your first query. | ||
| This script creates two small tables: one for employees and one for departments. | ||
| This Python script creates two small tables: one for employees and one for departments. | ||
| It joins the two tables on the DeptID column to show the name of the department | ||
| where each employee works. | ||
| ```python | ||
|
|
||
| from deephaven.TableTools import newTable, stringCol, intCol | ||
| from deephaven.conversion_utils import NULL_INT | ||
|
|
||
| left = newTable( | ||
| stringCol("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"), | ||
| intCol("DeptID", 31, 33, 33, 34, 34, NULL_INT), | ||
| stringCol("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None) | ||
| ) | ||
|
|
||
| right = newTable( | ||
| intCol("DeptID", 31, 33, 34, 35), | ||
| stringCol("DeptName", "Sales", "Engineering", "Clerical", "Marketing"), | ||
| stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111") | ||
| ) | ||
|
|
||
| t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone") | ||
| from deephaven import new_table | ||
| from deephaven.column import string_col, int_col | ||
| from deephaven.constants import NULL_INT | ||
| left = new_table([ | ||
| string_col("LastName", ["Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"]), | ||
| int_col("DeptID", [31, 33, 33, 34, 34, NULL_INT]), | ||
| string_col("Telephone", ["(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None]) | ||
| ]) | ||
| right = new_table([ | ||
| int_col("DeptID", [31, 33, 34, 35]), | ||
| string_col("DeptName", ["Sales", "Engineering", "Clerical", "Marketing"]), | ||
| string_col("Telephone", ["(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111"]) | ||
| ]) | ||
| t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone") | ||
| ``` | ||
|  | ||
|
|
@@ -237,15 +177,19 @@ t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone") | |
| * [Java API docs](https://deephaven.io/core/javadoc/) | ||
| * [Python API docs](https://deephaven.io/core/pydoc/) | ||
| ## Code Of Conduct | ||
| ## Contributing | ||
| See [CONTRIBUTING](./CONTRIBUTING.md) for full instructions on how to contribute to this project. | ||
| ### Code Of Conduct | ||
| This project has adopted the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/). | ||
| For more information see the [Code of Conduct](CODE_OF_CONDUCT.md) or contact [[email protected]](mailto:[email protected]) | ||
| with any additional questions or comments. | ||
| ## License | ||
| ### License | ||
| Copyright (c) Deephaven Data Labs. All rights reserved. | ||
| Provided under the [Deephaven Community License](LICENSE.md). | ||
| Provided under the [Deephaven Community License](LICENSE.md). | ||