diff --git a/README.md b/README.md index 3ac67d441a5..d5623c204a9 100644 --- a/README.md +++ b/README.md @@ -9,17 +9,9 @@ Deephaven includes an intuitive user experience and visualization tools. It can ingest data from a variety of sources, apply computation and analysis algorithms to that data, and build rich queries, dashboards, and representations with the results. -Deephaven Community Core is an open version of [Deephaven Enterprise](https://deephaven.io), +Deephaven Community Core is the open version of [Deephaven Enterprise](https://deephaven.io), which functions as the data backbone for prominent hedge funds, banks, and financial exchanges. -[![Join the chat at https://gitter.im/deephaven/deephaven](https://badges.gitter.im/deephaven/deephaven.svg)](https://gitter.im/deephaven/deephaven?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -![Build CI](https://github.com/deephaven/deephaven-core/actions/workflows/build-ci.yml/badge.svg?branch=main) -![Quick CI](https://github.com/deephaven/deephaven-core/actions/workflows/quick-ci.yml/badge.svg?branch=main) -![Docs CI](https://github.com/deephaven/deephaven-core/actions/workflows/docs-ci.yml/badge.svg?branch=main) -![Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/check-ci.yml/badge.svg?branch=main) -![Nightly Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/nightly-check-ci.yml/badge.svg?branch=main) -![Nightly Benchmarks](https://github.com/deephaven/deephaven-core/actions/workflows/nightly-benchmarks.yml/badge.svg?branch=main) - ## Supported Languages | Language | Server Application | Client Application | @@ -30,200 +22,148 @@ which functions as the data backbone for prominent hedge funds, banks, and finan | JavaScript | No | Yes | | Go | No | Yes | | R | No | Yes | -| gRPC | - | Yes | - -## Run Deephaven - -There are both server- and client-side applications available, as seen in the table above. -### Server-side +## Installation -Deephaven's server-side APIs allow you to connect directly to the Deephaven server and execute commands, write queries, and much more. +Deephaven Community Core can be used from both server- and client-side APIs. Server-side APIs allow users to connect directly to the Deephaven server and execute commands, write queries, and much more. Client-side APIs allow users to connect to a Deephaven server through a client application. -Most users will want to run Deephaven from pre-built images. It's the easiest way to configure, deploy, and use Deephaven. For detailed instructions, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart). +See the following documentation links for installation instructions and more: -Some Python users will want to use Deephaven entirely from Python. For instructions on starting and using Deephaven from Python, see [Quick start for pip-installed Deephaven](https://deephaven.io/core/docs/tutorials/quickstart-pip/). +- Python + - [Run from Docker](https://deephaven.io/core/docs/tutorials/quickstart/) + - [pip-installed](https://deephaven.io/core/docs/tutorials/quickstart/) +- [Groovy] + - [Run from Docker](https://deephaven.io/core/groovy/docs/tutorials/quickstart/) +- [Python client](https://pypi.org/project/pydeephaven/) +- [Java client](https://deephaven.io/core/docs/how-to-guides/java-client/) +- [JS client](https://deephaven.io/core/docs/reference/js-api/documentation/) +- [Go client](https://pkg.go.dev/github.com/deephaven/deephaven-core/go) +- [R client](https://github.com/deephaven/deephaven-core/blob/main/R/rdeephaven/README.md) -Developers interested in tinkering with and modifying source code should build from source. For detailed instructions on how to do this, see [Build and launch Deephaven](https://deephaven.io/core/docs/how-to-guides/launch-build). +Deephaven's client APIs use [gRPC](https://grpc.io/), [protobuf](https://github.com/deephaven/deephaven-core/tree/main/proto/proto-backplane-grpc/src/main/proto/deephaven/proto), [Apache Arrow Flight](https://arrow.apache.org/docs/format/Flight.html), and [Barrage](https://github.com/deephaven/barrage) to handle ticking data. Users who wish to build their own client APIs can use these tools to do so. -If you are not sure which of the two is right for you, use the pre-built images. +## Build from source -### Client-side +Users who wish to modify or contribute to this project should [build from source](https://deephaven.io/core/docs/how-to-guides/launch-build/) to run either the Python or Groovy server-side API. -Deephaven's client APIs allow you to connect to a Deephaven server and execute code. Documentation links are below: +Start by cloning this repository, and `cd` into the cloned repo. -- [Python](https://deephaven.io/core/client-api/python/) -- [Java/Groovy](https://deephaven.io/core/javadoc/) -- [C++](https://deephaven.io/core/client-api/cpp/) -- [JavaScript](https://deephaven.io/core/docs/reference/js-api/documentation/) -- [Go](https://pkg.go.dev/github.com/deephaven/deephaven-core/go) -- Documentation for R can be found with the `?` operator in R. +```bash +git clone https://github.com/deephaven/deephaven-core.git +cd deephaven-core +``` -### Docker Dependencies +### Required dependencies -Running Deephaven from pre-built Docker images requires a few software packages. +Building and running Deephaven requires a few software packages. | Package | Version | OS | | -------------- | ----------------------------- | ------------ | +| git | ^2.25.0 | All | +| java | >=11, <20 | All | | docker | ^20.10.8 | All | | docker-compose | ^1.29.0 | All | | Windows | 10 (OS build 20262 or higher) | Only Windows | | WSL | 2 | Only Windows | You can check if these packages are installed and functioning by running: -``` + +```bash +git version +java -version docker version docker-compose version docker run hello-world ``` -> :warning: **On Windows, all commands must be run inside a WSL 2 terminal.** +:::note -If any dependencies are missing or unsupported versions are installed, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart#prerequisites) for installation instructions. +Internally, the Java build process will use [Gradle Auto Provisioning](https://docs.gradle.org/current/userguide/toolchains.html#sec:provisioning) +to download and use the appropriate Java version for building and testing. +On Windows, all commands must be run inside a WSL 2 terminal. -### Create deployment +::: -A directory must be created to store files and mount points for your deployment. Here, we are using the `deephaven-deployment` directory. +### Python -You will need to `cd` into the deployment directory to launch or interact with the deployment. +Set up a virtual environment. ```bash -mkdir deephaven-deployment -cd deephaven-deployment +python -m venv /tmp/my-dh-venv +source /tmp/my-dh-venv/bin/activate ``` -> :warning: **Commands in the following sections for interacting with a deployment must be run from the deployment directory.** - -### Launch options - -Deephaven offers a variety of pre-made Docker Compose YAML files for different deployments. The following commands will launch Deephaven. Replace `` with that of the deployment from the table below that best suits your needs. +Then, build and install the wheel. ```bash -curl -O -docker compose pull -docker compose up -d +./gradlew py-server:assemble +pip install "py/server/build/wheel/deephaven_core--py3-non-any.whl[autocomplete] ``` -| Language | Example data | Additional packages | URL | -| Python | No | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/base/docker-compose.yml | -| Python | No | SciKit-Learn | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/SciKit-Learn/docker-compose.yml | -| Python | No | PyTorch | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml | -| Python | No | TensorFlow | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/TensorFlow/docker-compose.yml | -| Python | No | NLTK | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml | -| Python | Yes | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml | -| Python | Yes | SciKit-Learn | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml | -| Python | Yes | PyTorch | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml | -| Python | Yes | TensorFlow | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml | -| Python | Yes | NLTK | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml | -| Groovy | No | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml | -| Groovy | Yes | None | https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml | - +Where: -### Monitor logs +- `` is replaced by a deephaven-core [release](https://github.com/deephaven/deephaven-core/releases). +- Users who wish to use Deephaven without autocomplete should remove `[autocomplete]` from the `pip install` command. -The `-d` option to `docker-compose` causes the containers to run in the background, in detached mode. This option allows you to use your shell after Docker launches the containers. - -Since the container is running detached, you will not see any logs. However, you can follow the logs by running: +Lastly, build. ```bash -docker-compose logs -f +./gradlew server-jetty-app:run ``` -Use CTRL+C to stop monitoring the logs and return to a prompt. - -### Authentication - -Deephaven, by default, comes stock with pre-shared key to authenticate any users trying to access it. If no key is set, a randomly generated key is printed to the Docker logs. The key can be found by monitoring the logs. It will look like this: +### Groovy/Java -``` -================================================================================ -Superuser access through pre-shared key is enabled - use 1c3opvgzl5scm to connect -Connect automatically to Web UI with http://localhost:10000/?psk=1c3opvgzl5scm -================================================================================ -``` - -You can also `grep` for the key in the logs: +This single command will build and run the Deephaven Groovy server-side API from source. ```bash -docker compose logs -f | grep "access through pre-shared key" +./gradlew server-jetty-app:run ``` -For more information on setting your own pre-shared key, see [How to configure and use pre-shared key authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-psk/). - -For information on enabling anonymous authentication, see [How to enable anonymous authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-anon/). - -### Shutdown - -The deployment can be brought down by running: - -```bash -docker-compose down -``` - -### Manage example data - -[Deephaven's examples repository](https://github.com/deephaven/examples) contains data sets that are useful when learning -to use Deephaven. These data sets are used extensively in Deephaven's documentation and are needed to run some examples. [Deephaven's examples repository](https://github.com/deephaven/examples) contains documentation on the available data sets and how to manage them. - -If you have chosen a deployment with example data, the example data sets will be downloaded. Production deployments containing your own data will not need the example data sets. - +### Authentication -To upgrade a deployment to the latest example data, run: +Deephaven, by default, uses pre-shared key authentication to authenticate against unauthorized access. Users will be prompted for a key when connecting to an instance of the server. Unless otherwise specified, a new randomly generated key will be used each time the server is started. The key is printed to the Docker logs. To search the logs for the key, run: ```bash -docker-compose run examples download +docker compose logs -f | grep "access through pre-shared key" ``` -To see what other example data management commands are available, run: +To set the key, add `-Ppsk` to the build command: ```bash -docker-compose run examples +./gradlew server-jetty-app:run -Ppsk=My-Password ``` -If your deployment does not have example data, these commands will fail with `ERROR: No such service`. - -### Build from source - -If you are interested in tinkering and modifying source code for your own Deephaven deployment, or contributing to the project, see this [README](https://github.com/deephaven/deephaven-core/blob/main/server/jetty-app/README.md) for build instructions. There are several [required dependencies](https://deephaven.io/core/docs/how-to-guides/launch-build/#required-dependencies) to build and launch from source code. +### Connect to the server -Additionally, see our [code of conduct](https://github.com/deephaven/deephaven-core/blob/main/CODE_OF_CONDUCT.md) and [contributing guide](https://github.com/deephaven/deephaven-core/blob/main/CONTRIBUTING.md). - -## Run Deephaven IDE - -Once Deephaven is running, you can launch a Deephaven IDE in your web browser. Deephaven IDE allows you to interactively analyze data. - -- If Deephaven is running locally, navigate to [http://localhost:10000/ide/](http://localhost:10000/ide/). -- If Deephaven is running remotely, navigate to `http://:10000/ide/`, where `` is the address of the machine Deephaven is running on. - -![alt_text](docs/images/ide_startup.png "Deephaven IDE") +Deephaven is run from a web browser, and can be connected to via `http://localhost:10000/ide`. ## First query From the Deephaven IDE, you can perform your first query. -This script creates two small tables: one for employees and one for departments. +This Python script creates two small tables: one for employees and one for departments. It joins the two tables on the DeptID column to show the name of the department where each employee works. ```python - -from deephaven.TableTools import newTable, stringCol, intCol -from deephaven.conversion_utils import NULL_INT - -left = newTable( - stringCol("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"), - intCol("DeptID", 31, 33, 33, 34, 34, NULL_INT), - stringCol("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None) - ) - -right = newTable( - intCol("DeptID", 31, 33, 34, 35), - stringCol("DeptName", "Sales", "Engineering", "Clerical", "Marketing"), - stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111") - ) - -t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone") +from deephaven import new_table +from deephaven.column import string_col, int_col +from deephaven.constants import NULL_INT + +left = new_table([ + string_col("LastName", ["Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"]), + int_col("DeptID", [31, 33, 33, 34, 34, NULL_INT]), + string_col("Telephone", ["(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None]) + ]) + +right = new_table([ + int_col("DeptID", [31, 33, 34, 35]), + string_col("DeptName", ["Sales", "Engineering", "Clerical", "Marketing"]), + string_col("Telephone", ["(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111"]) + ]) + +t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone") ``` ![alt_text](docs/images/ide_first_query.png "Deephaven IDE First Query") @@ -237,15 +177,19 @@ t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone") * [Java API docs](https://deephaven.io/core/javadoc/) * [Python API docs](https://deephaven.io/core/pydoc/) -## Code Of Conduct +## Contributing + +See [CONTRIBUTING](./CONTRIBUTING.md) for full instructions on how to contribute to this project. + +### Code Of Conduct This project has adopted the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/). For more information see the [Code of Conduct](CODE_OF_CONDUCT.md) or contact [opencode@deephaven.io](mailto:opencode@deephaven.io) with any additional questions or comments. -## License +### License Copyright (c) Deephaven Data Labs. All rights reserved. -Provided under the [Deephaven Community License](LICENSE.md). +Provided under the [Deephaven Community License](LICENSE.md). \ No newline at end of file