From fb5aa43cb60448b925f9465e19baefb877286f72 Mon Sep 17 00:00:00 2001 From: wanghui42 <105700158+wanghui42@users.noreply.github.com> Date: Fri, 2 Feb 2024 11:40:44 +0800 Subject: [PATCH] add the AINode deploy doc (#162) --- .../Master/API/Programming-TsFile-API.md | 529 ------------------ .../Deployment-Guide_timecho.md | 315 ++++++++++- .../Deployment-Guide_timecho.md | 312 ++++++++++- 3 files changed, 622 insertions(+), 534 deletions(-) delete mode 100644 src/UserGuide/Master/API/Programming-TsFile-API.md diff --git a/src/UserGuide/Master/API/Programming-TsFile-API.md b/src/UserGuide/Master/API/Programming-TsFile-API.md deleted file mode 100644 index 4ee7b4dc..00000000 --- a/src/UserGuide/Master/API/Programming-TsFile-API.md +++ /dev/null @@ -1,529 +0,0 @@ - - -# TsFile API - -`TsFile` is a file format of time series used in IoTDB. -This document introduces the usage of this file format. - -## TsFile library Installation - -There are several ways to use TsFile in your own project. - -* Build from source and use the jars: - ```shell - git clone https://github.com/apache/iotdb.git - cd iotdb-core/tsfile - mvn clean package -DskipTests - ``` - Then, all the jars are located in the folder named `target/`. Import `target/tsfile-0.12.0-jar-with-dependencies.jar` into your project. -* Build from source and use as maven dependency: - Compile source codes and deploy to your local repository in three steps: - * Get the source codes - ```shell - git clone https://github.com/apache/iotdb.git - cd iotdb-core/tsfile - mvn clean install -DskipTests - ``` - * Add the following dependency into your project: - ```xml - - org.apache.iotdb - tsfile - 1.0.0 - - ``` -* Download the convenience binaries available from Maven-Central: - * (If you want to reference the latest SNAPSHOT versions, you need to execute this step) - Add the `Apache Snapshot Repository` to your projects main `pom.xml`: - ```xml - - - apache.snapshots - Apache Development Snapshot Repository - https://repository.apache.org/content/repositories/snapshots/ - - false - - - true - - - - ``` - Alternately, find or create your maven `settings.xml` located at: `${username}\.m2\settings.xml`, add this `` to ``: - ```xml - - allow-snapshots - - true - - - - apache.snapshots - Apache Development Snapshot Repository - https://repository.apache.org/content/repositories/snapshots/ - - false - - - true - - - - - ``` - * Then add dependency into your project: - ```xml - - org.apache.iotdb - tsfile - 1.0.0 - - ``` - -## TsFile Usage - -This section demonstrates the detailed usages of TsFile. - -Time-series data is considered as a sequence of quadruples. A quadruple is defined as (device, measurement, time, value). - -* **measurement**: A physical or formal measurement that a time-series data takes, e.g. the temperature of a city, the sales number of some goods or the speed of a train at different times. - As a traditional sensor (like a thermometer) also produces a single measurement which we can use to create a time-series, we will use measurement and sensor interchangeably below. - -* **device**: A device refers to an entity that produces one or multiple measurements (producing multiple time-series), e.g., - a running train monitors its speed, oil meter, miles it has run, current passengers. - Each is persisted to a time-series dataset. - -* **Row of Data**: In many industrial applications, a device normally contains more than one sensor and these sensors may have values at the same timestamp, which is called `row of data`. - Formally, a `row of data` consists of a `device_id`, a `timestamp` which indicates the milliseconds since `January 1, 1970, 00:00:00`, and several data pairs composed of `measurement_id` and corresponding `value`. - All data pairs in a line of data belong to the same `device_id` and have the same timestamp. - If one of the `measurements` does not have a `value` in the `timestamp`, a space is used instead (Actually, TsFile does not actually store `null` values). - Its format is shown as follows: - ``` - device_id, timestamp, ... - ``` - An example is illustrated as follows. - In this example, the data type of two measurements are `INT32`, `FLOAT` respectively. - ``` - device_1, 1490860659000, m1, 10, m2, 12.12 - ``` - -### Write TsFile - -A `TsFile` is generated by the following steps (The complete code is given in [TsFile examples module](https://github.com/apache/iotdb/tree/master/example/tsfile)): - -1. Construct a `TsFileWriter` instance. -2. Define the `Schema` for the `TSFile` (However a pre-defined Schema can also be passed directly to the constructor in step 1). -3. Write data to the `TsFileWriter`. -4. Close the `TsFileWriter` (When using a Java try-with-resources block, Java will take care of closing the TsFileWriter). - -#### Construct an `TsFileWriter` instance. - -Here are the available constructors: -* Without pre-defined schema: - ```java - TsFileWriter(File file) throws IOException - ``` -* With pre-defined schema: - ```java - TsFileWriter(File file, Schema schema) throws IOException - ``` -* Providing a `TsFileOutput` instead of a File with a schema (useful when using the HDFS file system as `TsFileOutput` can be an instance of class `HDFSOutput`): - ```java - TsFileWriter(TsFileOutput output, Schema schema) throws IOException - ``` -* If you want to set some TSFile configuration on your own, you could use param `config`. For example: - ```java - TSFileConfig conf = new TSFileConfig(); - conf.setTSFileStorageFs("HDFS"); - TsFileWriter tsFileWriter = new TsFileWriter(file, schema, conf); - ``` - In this example, data files will be stored in HDFS, instead of local file system. - If you'd like to store data files in local file system, you can use `conf.setTSFileStorageFs("LOCAL")`, which is also the default config. - You can configure the `ip` and `rpc port` of your HDFS by setting `config.setHdfsIp(...)` and `config.setHdfsPort(...)`. The default ip is `localhost` and default rpc port is `9000`. - -**Parameters:** - -* file : The TsFile to write -* schema : The file schemas, will be introduced in next part. -* config : The config of TsFile. - -#### Construct a `Schema` instance. - -The class `Schema` contains a map whose key is the name of one measurement schema, and the value is the schema itself. -Here are the most important methods: -```java -// Create an empty Schema or from an existing map -public Schema() -public Schema(Map measurements) -// Use this two interfaces to add measurements -public void registerMeasurement(MeasurementSchema descriptor) -public void registerMeasurements(Map measurements) -// Some useful getter and checker -public TSDataType getMeasurementDataType(String measurementId) -public MeasurementSchema getMeasurementSchema(String measurementId) -public Map getAllMeasurementSchema() -public boolean hasMeasurement(String measurementId) -``` -The class `MeasurementSchema` contains the information of one measurement, there are several constructors: - -```java -public MeasurementSchema(String measurementId, TSDataType type, TSEncoding encoding) -public MeasurementSchema(String measurementId, TSDataType type, TSEncoding encoding, CompressionType compressionType) -public MeasurementSchema(String measurementId, TSDataType type, TSEncoding encoding, CompressionType compressionType, Map props) -``` -**Parameters:** - -* measurementID: The name of this measurement, typically the name of the sensor. -* type: The data type, now support six types: `BOOLEAN`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `TEXT`; -* encoding: The data encoding. -* compressionType: The data compression type. -* props: A map of properties for special data types, such as `max_point_number` for `FLOAT` and `DOUBLE`, `max_string_length` for `TEXT`. Use as string pairs into a map such as ("max_point_number", "3"). - -> **Notice:** Although one measurement name can be used in multiple deltaObjects, the properties cannot be changed. I.e. it is not allowed to add one measurement name for multiple times with different type or encoding. Here is a bad example: - -```java -List measurementSchemas = new ArrayList<>(); -// The measurement "sensor_1" is float type -measurementSchemas.add(new MeasurementSchema("sensor_1", TSDataType.FLOAT, TSEncoding.RLE)); -measurementSchemas.add(new MeasurementSchema("sensor_1", TSDataType.INT32, TSEncoding.RLE)); -``` -#### Insert and write data - -Use this interface to create a new `TSRecord`(a timestamp and device pair). - -```java - TSRecord tsRecord = new TSRecord(time, deviceId); -``` -Then create a `DataPoint`(a measurement and value pair), and use the `addTuple` method to add the `DataPoint` to the current `TsRecord`. -```java - DataPoint dPoint = new LongDataPoint("sensor_1", 42); - tsRecord.addTuple(dPoint); -``` -> Notice: there are implementations of `DataPoint` for each of IoTDBs supported data types: `BooleanDataPoint`, `DoubleDataPoint`, `FloatDataPoint`, `IntDataPoint`, `LongDataPoint` and `StringDataPoint`. - -As soon as the TSRecord is finished, write it to file with the following command: -```java - tsFileWriter.write(tsRecord); -``` - -#### Call `close` to finish this writing process - -```java - tsFileWriter.close(); -``` - -We are also able to write data into a closed TsFile. - -1. Use `ForceAppendTsFileWriter` to open a closed file. - - ```java - public ForceAppendTsFileWriter(File file) throws IOException - ``` - -2. Call `doTruncate()` to truncate the part of Metadata - -3. Then use `ForceAppendTsFileWriter` to construct a new `TsFileWriter` - -```java -public TsFileWriter(TsFileIOWriter fileWriter) throws IOException -``` -Please note, we should redo the step of adding measurements before writing new data to the TsFile. - -### Example for writing a TsFile - -You should install TsFile to your local maven repository. - -```shell -mvn clean install -pl iotdb-core/tsfile -am -DskipTests -``` - -You could write a TsFile by constructing **TSRecord** if you have the **non-aligned** (e.g. not all sensors contain values) time series data. - -A more thorough example can be found at `/example/tsfile/src/main/java/org/apache/iotdb/tsfile/TsFileWriteWithTSRecord.java` - -You could write a TsFile by constructing **Tablet** if you have the **aligned** time series data. - -A more thorough example can be found at `/example/tsfile/src/main/java/org/apache/iotdb/tsfile/TsFileWriteWithTablet.java` - -You could write data into a closed TsFile by using **ForceAppendTsFileWriter**. - -A more thorough example can be found at `/example/tsfile/src/main/java/org/apache/iotdb/tsfile/TsFileForceAppendWrite.java` - -### Interface for Reading TsFile - -* Definition of Path - -A path is a dot-separated string which uniquely identifies a time-series in TsFile, e.g., "root.area_1.device_1.sensor_1". -The last section "sensor_1" is called "measurementId" while the remaining parts "root.area_1.device_1" is called deviceId. -As mentioned above, the same measurement in different devices has the same data type and encoding, and devices are also unique. - -In read interfaces, The parameter `paths` indicates the measurements to be selected. - -Path instance can be easily constructed through the class `Path`. For example: - -```java -Path p = new Path("device_1.sensor_1"); -``` - -We will pass an ArrayList of paths for final query call to support multiple paths. - -```java -List paths = new ArrayList(); -paths.add(new Path("device_1.sensor_1")); -paths.add(new Path("device_1.sensor_3")); -``` - -> **Notice:** When constructing a Path, the format of the parameter should be a dot-separated string, the last part will - be recognized as measurementId while the remaining parts will be recognized as deviceId. - - -* Definition of Filter - - * Usage Scenario -Filter is used in TsFile reading process to select data satisfying one or more given condition(s). - - * IExpression -The `IExpression` is a filter expression interface and it will be passed to our final query call. -We create one or more filter expressions and may use binary filter operators to link them to our final expression. - -* **Create a Filter Expression** - - There are two types of filters. - - * TimeFilter: A filter for `time` in time-series data. - ``` - IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter); - ``` - Use the following relationships to get a `TimeFilter` object (value is a long int variable). - - |Relationship|Description| - |---|---| - |TimeFilter.eq(value)|Choose the time equal to the value| - |TimeFilter.lt(value)|Choose the time less than the value| - |TimeFilter.gt(value)|Choose the time greater than the value| - |TimeFilter.ltEq(value)|Choose the time less than or equal to the value| - |TimeFilter.gtEq(value)|Choose the time greater than or equal to the value| - |TimeFilter.notEq(value)|Choose the time not equal to the value| - |TimeFilter.not(TimeFilter)|Choose the time not satisfy another TimeFilter| - - * ValueFilter: A filter for `value` in time-series data. - - ``` - IExpression valueFilterExpr = new SingleSeriesExpression(Path, ValueFilter); - ``` - The usage of `ValueFilter` is the same as using `TimeFilter`, just to make sure that the type of the value - equal to the measurement's(defined in the path). - -* **Binary Filter Operators** - - Binary filter operators can be used to link two single expressions. - - * BinaryExpression.and(Expression, Expression): Choose the value satisfy for both expressions. - * BinaryExpression.or(Expression, Expression): Choose the value satisfy for at least one expression. - -Filter Expression Examples - -* **TimeFilterExpression Examples** - - ```java - IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.eq(15)); // series time = 15 - ``` -``` - ```java - IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.ltEq(15)); // series time <= 15 -``` -```java - IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.lt(15)); // series time < 15 -``` - ```java -IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.gtEq(15)); // series time >= 15 - ``` - ```java - IExpression timeFilterExpr = new GlobalTimeExpression(TimeFilter.notEq(15)); // series time != 15 -``` - ```java - IExpression timeFilterExpr = BinaryExpression.and( - new GlobalTimeExpression(TimeFilter.gtEq(15L)), - new GlobalTimeExpression(TimeFilter.lt(25L))); // 15 <= series time < 25 -``` - ```java - IExpression timeFilterExpr = BinaryExpression.or( - new GlobalTimeExpression(TimeFilter.gtEq(15L)), - new GlobalTimeExpression(TimeFilter.lt(25L))); // series time >= 15 or series time < 25 - ``` -* Read Interface - -First, we open the TsFile and get a `ReadOnlyTsFile` instance from a file path string `path`. - -```java -TsFileSequenceReader reader = new TsFileSequenceReader(path); - -ReadOnlyTsFile readTsFile = new ReadOnlyTsFile(reader); -``` -Next, we prepare the path array and query expression, then get final `QueryExpression` object by this interface: - -```java -QueryExpression queryExpression = QueryExpression.create(paths, statement); -``` - -The ReadOnlyTsFile class has two `query` method to perform a query. -* **Method 1** - - ```java - public QueryDataSet query(QueryExpression queryExpression) throws IOException - ``` - -* **Method 2** - - ```java - public QueryDataSet query(QueryExpression queryExpression, long partitionStartOffset, long partitionEndOffset) throws IOException - ``` - - This method is designed for advanced applications such as the TsFile-Spark Connector. - - * **params** : For method 2, two additional parameters are added to support partial query: - * ```partitionStartOffset```: start offset for a TsFile - * ```partitionEndOffset```: end offset for a TsFile - - > **What is Partial Query ?** - > - > In some distributed file systems(e.g. HDFS), a file is split into severval parts which are called "Blocks" and stored in different nodes. Executing a query paralleled in each nodes involved makes better efficiency. Thus Partial Query is needed. Paritial Query only selects the results stored in the part split by ```QueryConstant.PARTITION_START_OFFSET``` and ```QueryConstant.PARTITION_END_OFFSET``` for a TsFile. - -* QueryDataset Interface - -The query performed above will return a `QueryDataset` object. - -Here's the useful interfaces for user. - - * `bool hasNext();` - - Return true if this dataset still has elements. - * `List getPaths()` - - Get the paths in this data set. - * `List getDataTypes();` - - Get the data types. The class TSDataType is an enum class, the value will be one of the following: - - BOOLEAN, - INT32, - INT64, - FLOAT, - DOUBLE, - TEXT; - * `RowRecord next() throws IOException;` - - Get the next record. - - The class `RowRecord` consists of a `long` timestamp and a `List` for data in different sensors, - we can use two getter methods to get them. - - ```java - long getTimestamp(); - List getFields(); - ``` - - To get data from one Field, use these methods: - - ```java - TSDataType getDataType(); - Object getObjectValue(); - ``` - - - -### Example for reading an existing TsFile - - -You should install TsFile to your local maven repository. - - -A more thorough example with query statement can be found at -`/tsfile/example/src/main/java/org/apache/iotdb/tsfile/TsFileRead.java` - -```java -package org.apache.iotdb.tsfile; -import java.io.IOException; -import java.util.ArrayList; -import org.apache.iotdb.tsfile.read.ReadOnlyTsFile; -import org.apache.iotdb.tsfile.read.TsFileSequenceReader; -import org.apache.iotdb.tsfile.read.common.Path; -import org.apache.iotdb.tsfile.read.expression.IExpression; -import org.apache.iotdb.tsfile.read.expression.QueryExpression; -import org.apache.iotdb.tsfile.read.expression.impl.BinaryExpression; -import org.apache.iotdb.tsfile.read.expression.impl.GlobalTimeExpression; -import org.apache.iotdb.tsfile.read.expression.impl.SingleSeriesExpression; -import org.apache.iotdb.tsfile.read.filter.TimeFilter; -import org.apache.iotdb.tsfile.read.filter.ValueFilter; -import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet; - -/** - * The class is to show how to read TsFile file named "test.tsfile". - * The TsFile file "test.tsfile" is generated from class TsFileWrite. - * Run TsFileWrite to generate the test.tsfile first - */ -public class TsFileRead { - private static void queryAndPrint(ArrayList paths, ReadOnlyTsFile readTsFile, IExpression statement) - throws IOException { - QueryExpression queryExpression = QueryExpression.create(paths, statement); - QueryDataSet queryDataSet = readTsFile.query(queryExpression); - while (queryDataSet.hasNext()) { - System.out.println(queryDataSet.next()); - } - System.out.println("------------"); - } - - public static void main(String[] args) throws IOException { - - // file path - String path = "test.tsfile"; - - // create reader and get the readTsFile interface - TsFileSequenceReader reader = new TsFileSequenceReader(path); - ReadOnlyTsFile readTsFile = new ReadOnlyTsFile(reader); - // use these paths(all sensors) for all the queries - ArrayList paths = new ArrayList<>(); - paths.add(new Path("device_1.sensor_1")); - paths.add(new Path("device_1.sensor_2")); - paths.add(new Path("device_1.sensor_3")); - - // no query statement - queryAndPrint(paths, readTsFile, null); - - //close the reader when you left - reader.close(); - } -} -``` - - - -## Change TsFile Configuration - -```java -TSFileConfig config = TSFileDescriptor.getInstance().getConfig(); -config.setXXX(); -``` - - - diff --git a/src/UserGuide/Master/Deployment-and-Maintenance/Deployment-Guide_timecho.md b/src/UserGuide/Master/Deployment-and-Maintenance/Deployment-Guide_timecho.md index 73aac39d..40ff39ae 100644 --- a/src/UserGuide/Master/Deployment-and-Maintenance/Deployment-Guide_timecho.md +++ b/src/UserGuide/Master/Deployment-and-Maintenance/Deployment-Guide_timecho.md @@ -76,9 +76,6 @@ Users can start IoTDB standalone mode by the start-standalone script under the s > sbin\start-standalone.bat ``` -Note: Currently, To run standalone mode, you need to ensure that all addresses are set to 127.0.0.1, If you need to access the IoTDB from a machine different from the one where the IoTDB is located, please change the configuration item `dn_rpc_address` to the IP of the machine where the IoTDB lives. And replication factors set to 1, which is by now the default setting. -Besides, it's recommended to use SimpleConsensus in this mode, since it brings additional efficiency. - ## Cluster deployment(Cluster management tool) The IoTDB cluster management tool is an easy-to-use operation and maintenance tool (enterprise version tool). @@ -1166,4 +1163,314 @@ Run the remove-datanode script on an active DataNode: ### FAQ -See [FAQ](https://iotdb.apache.org/UserGuide/Master/FAQ/FAQ-for-cluster-setup.html). \ No newline at end of file +See [FAQ](https://iotdb.apache.org/UserGuide/Master/FAQ/FAQ-for-cluster-setup.html). + +## AINode deployment + +### Installation environment + +#### Recommended Operating System + +Ubuntu, CentOS, MacOS + +#### Runtime Environment + +AINode currently requires Python 3.8 or higher with pip and venv tools. + +For networked environments, AINode creates a virtual environment and downloads runtime dependencies automatically, no additional configuration is needed. + +In case of a non-networked environment, you can download it from https://cloud.tsinghua.edu.cn/d/4c1342f6c272439aa96c/to get the required dependencies and install them offline. + +### Installation steps + +Users can download the AINode software installation package, download and unzip it to complete the installation of AINode. You can also download the source code from the code repository and compile it to get the installation package. + +### Software directory structure + +After downloading and extracting the software package, you can get the following directory structure + +```Shell +|-- apache-iotdb-AINode-bin + |-- lib # package binary executable with environment dependencies + |-- conf # store configuration files + - iotdb-AINode.properties + |-- sbin # AINode related startup scripts + - start-AINode.sh + - start-AINode.bat + - stop-AINode.sh + - stop-AINode.bat + - remove-AINode.sh + - remove-AINode.bat + |-- licenses + - LICENSE + - NOTICE + - README.md + - README_ZH.md + - RELEASE_NOTES.md +``` + +- **lib:** AINode's compiled binary executable and related code dependencies. +- **conf:** contains AINode's configuration items, specifically the following configuration items +- **sbin:** AINode's runtime script, which can start, remove and stop AINode. + +### Start AINode + +After completing the deployment of Seed-ConfigNode, you can add an AINode node to support the model registration and inference functions. After specifying the information of IoTDB cluster in the configuration item, you can execute the corresponding commands to start AINode and join the IoTDB cluster. + +Note: Starting AINode requires that the system environment contains a Python interpreter of 3.8 or above as the default interpreter, so users should check whether the Python interpreter exists in the environment variables and can be directly invoked through the `python` command before using it. + +#### Direct Start + +After obtaining the installation package files, you can directly start AINode for the first time. + +The startup commands on Linux and MacOS are as follows: + +```Shell +> bash sbin/start-AINode.sh +``` + +The startup command on windows is as follows: + +```Shell +> sbin\start-AINode.bat +``` + +If start AINode for the first time and do not specify the path to the interpreter, the script will create a new venv virtual environment in the root directory of the program using the system Python interpreter, and install the third-party dependencies of AINode and the main program of AINode in this environment automatically and successively. **This process will generate a virtual environment of about 1GB in size, so please reserve space for installation**. On subsequent startups, if the path to the interpreter is not specified, the script will automatically look for the newly created venv environment above and start AINode without having to install the program and dependencies repeatedly. + +Note that it is possible to activate reinstall with -r if you wish to force a reinstall of AINode proper on a certain startup, this parameter will reinstall AINode based on the files under lib. + +Linux和MacOS： + +```Shell +> bash sbin/start-AINode.sh -r +``` + +Windows： + +```Shell +> sbin\start-AINode.bat -r +``` + +For example, a user replaces a newer version of the AINode installer in the lib, but the installer is not installed in the user's usual environment. In this case, you need to add the -r option at startup to instruct the script to force a reinstallation of the main AINode program in the virtual environment to update the version. + +#### Specify a customized virtual environment + +When starting AINode, you can specify a virtual environment interpreter path to install the AINode main program and its dependencies to a specific location. Specifically, you need to specify the value of the parameter ain_interpreter_dir. + +Linux and MacOS： + +```Shell +> bash sbin/start-AINode.sh -i xxx/bin/python +``` + +Windows： + +```Shell +> sbin\start-AINode.bat -i xxx\Scripts\python.exe +``` + +When specifying the Python interpreter please enter the address of the **executable file** of the Python interpreter in the virtual environment. Currently AINode **supports virtual environments such as venv, ****conda****, etc.** **Inputting the system Python interpreter as the installation location** is not supported. In order to ensure that scripts are recognized properly, please **use absolute paths whenever possible**! + +#### Join the cluster + +The AINode startup process automatically adds the new AINode to the IoTDB cluster. After starting the AINode you can verify that the node was joined successfully by entering the SQL for the cluster query in IoTDB's cli command line. +```Shell +IoTDB> show cluster ++------+----------+-------+---------------+------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710|UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730|UNKNOWN|190e303-dev| +| 2| AINode|Running| 127.0.0.1| 10810|UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------+-----------+ + +IoTDB> show cluster details ++------+----------+-------+---------------+------------+-------------------+----------+-------+-------+-------------------+-----------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort|RpcAddress|RpcPort|MppPort|SchemaConsensusPort|DataConsensusPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------------------+----------+-------+-------+-------------------+-----------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710| 10720| | | | | |UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730| | 0.0.0.0| 6667| 10740| 10750| 10760|UNKNOWN|190e303-dev| +| 2| AINode|Running| 127.0.0.1| 10810| | 0.0.0.0| 10810| | | |UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------------------+----------+-------+-------+-------------------+-----------------+-------+-----------+ + +IoTDB> show AINodes ++------+-------+----------+-------+ +|NodeID| Status|RpcAddress|RpcPort| ++------+-------+----------+-------+ +| 2|Running| 127.0.0.1| 10810| ++------+-------+----------+-------+ +``` + +### Remove AINode + +When it is necessary to move an already connected AINode out of the cluster, the corresponding removal script can be executed. + +The commands on Linux and MacOS are as follows: + +```Shell +> bash sbin/remove-AINode.sh +``` + +The startup command on windows is as follows: + +```Shell +> sbin/remove-AINode.bat +``` + +After removing the node, information about the node will not be available. + +```Shell +IoTDB> show cluster ++------+----------+-------+---------------+------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710|UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730|UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------+-----------+ +``` + +In addition, if the location of the AINode installation was previously customized, then the remove script should be called with the corresponding path as an argument: + +Linux and MacOS： + +```Shell +> bash sbin/remove-AINode.sh -i xxx/bin/python +``` + +Windows： + +```Shell +> sbin\remove-AINode.bat -i 1 xxx\Scripts\python.exe +``` + +Similarly, script parameters that are persistently modified in the env script will also take effect when the removal is performed. + +If a user loses a file in the data folder, AINode may not be able to remove itself locally, and requires the user to specify the node number, address and port number for removal, in which case we support the user to enter parameters for removal as follows + +Linux and MacOS： + +```Shell +> bash sbin/remove-AINode.sh -t /: +``` + +Windows： + +```Shell +> sbin\remove-AINode.bat -t /: +``` + +### Stop AINode + +If you need to stop a running AINode node, execute the appropriate shutdown script. + +The commands on Linux and MacOS are as follows: + +``` Shell. +> bash sbin/stop-AINode.sh +``` + +The startup command on windows is as follows: + +```Shell +> sbin/stop-AINode.bat +``` + +At this point the exact state of the node is not available and the corresponding management and reasoning functions cannot be used. If you need to restart the node, just execute the startup script again. + +```Shell +IoTDB> show cluster ++------+----------+-------+---------------+------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710|UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730|UNKNOWN|190e303-dev| +| 2| AINode|UNKNOWN| 127.0.0.1| 10790|UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------+-----------+ +``` + +### Script parameter details + +Two parameters are supported during AINode startup, and their specific roles are shown below: + +| **Name** | **Action Script** | Tag | **Description** | **Type** | **Default Value** | Input Method | +| ------------------- | ---------------- | ---- | ------------------------------------------------------------ | -------- | ---------------- | --------------------- | +| ain_interpreter_dir | start remove env | -i | The path to the interpreter of the virtual environment in which AINode is installed; absolute paths are required. | String | Read environment variables by default | Input on call + persistent modifications | +| ain_remove_target | remove stop | -t | AINode shutdown can specify the Node ID, address, and port number of the target AINode to be removed, in the format of `/:` | String | Null | Input on call | +| ain_force_reinstall | start remove env | -r | This script checks the version of the AINode installation, and if it does, it forces the installation of the whl package in lib if the version is not correct. | Bool | false | Input on call | +| ain_no_dependencies | start remove env | -n | Specifies whether to install dependencies when installing AINode, if so only the main AINode program will be installed without dependencies. | Bool | false | Input on call | + +Besides passing in the above parameters when executing the script as described above, it is also possible to modify some of the parameters persistently in the `AINode-env.sh` and `AINode-env.bat` scripts in the `conf` folder. + +`AINode-env.sh`： + +```Bash +# The defaulte venv environment is used if ain_interpreter_dir is not set. Please use absolute path without quotation mark +# ain_interpreter_dir= +``` + +`AINode-env.bat`： + +```Plain +@REM The defaulte venv environment is used if ain_interpreter_dir is not set. Please use absolute path without quotation mark +@REM set ain_interpreter_dir= +``` + +Uncomment the corresponding line after writing the parameter value and save it to take effect the next time you execute the script. + +### AINode configuration items + +AINode supports modifying some necessary parameters. The following parameters can be found in the `conf/iotdb-AINode.properties` file and modified for persistence: + +| **Name** | **Description** | **Type** | **Default Value** | **Modified Mode of Effect** | +| --------------------------- | ------------------------------------------------------------ | -------- | ------------------ | ---------------------------- | +| ain_target_config_node_list | ConfigNode address registered at AINode startup | String | 10710 | Only allow to modify before the first startup | +| ain_inference_rpc_address | Addresses where AINode provides services and communications | String | 127.0.0.1 | Effective after reboot | +| ain_inference_rpc_port | AINode provides services and communication ports | String | 10810 | Effective after reboot | +| ain_system_dir | AINode metadata storage path, the starting directory of the relative path is related to the operating system, it is recommended to use the absolute path. | String | data/AINode/system | Effective after reboot | +| ain_models_dir | AINode stores the path to the model file. The starting directory of the relative path is related to the operating system, and an absolute path is recommended. | String | data/AINode/models | Effective after reboot | +| ain_logs_dir | The path where AINode stores the logs. The starting directory of the relative path is related to the operating system, and it is recommended to use the absolute path. | String | logs/AINode | Effective after reboot | + +### Frequently Asked Questions + +1. **Not found venv module error when starting AINode** + +When starting AINode using the default method, a python virtual environment is created in the installation package directory and dependencies are installed, thus requiring the installation of the venv module. Generally speaking, python 3.8 and above will come with venv, but for some systems that come with python environment may not fulfill this requirement. There are two solutions when this error occurs (either one or the other): + +- Install venv module locally, take ubuntu as an example, you can run the following command to install the venv module that comes with python. Or install a version of python that comes with venv from the python website. + +```SQL +apt-get install python3.8-venv +``` + +- Specify the path to an existing python interpreter as the AINode runtime environment via -i when running the startup script, so that you no longer need to create a new virtual environment. + +2. **Compiling the python environment in CentOS7** + +The new environment in centos7 (comes with python3.6) does not meet the requirements to start mlnode, you need to compile python3.8+ by yourself (python is not provided as a binary package in centos7) + +- Install OpenSSL + +> Currently Python versions 3.6 to 3.9 are compatible with OpenSSL 1.0.2, 1.1.0, and 1.1.1. + +Python requires that we have OpenSSL installed on our system, which can be found at https://stackoverflow.com/questions/56552390/how-to-fix-ssl-module-in-python-is-not-available-in-centos + +- Installation and compilation of python + +Download the installation package from the official website and extract it using the following specifications + +```SQL +wget https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz +tar -zxvf Python-3.8.1.tgz +``` + +Compile and install the corresponding python packages. + +```SQL +./configure prefix=/usr/local/python3 -with-openssl=/usr/local/openssl +make && make install +``` + +1. **Windows compilation problem like "error: Microsoft Visual** **C++** **14.0 or greater is required..." compilation problem** on windows. + +The corresponding error is usually caused by an insufficient version of c++ or setuptools, you can find the appropriate solution at https://stackoverflow.com/questions/44951456/pip-error-microsoft-visual-c-14-0-is-required +you can find a suitable solution there. \ No newline at end of file diff --git a/src/UserGuide/latest/Deployment-and-Maintenance/Deployment-Guide_timecho.md b/src/UserGuide/latest/Deployment-and-Maintenance/Deployment-Guide_timecho.md index 44d133bd..40ff39ae 100644 --- a/src/UserGuide/latest/Deployment-and-Maintenance/Deployment-Guide_timecho.md +++ b/src/UserGuide/latest/Deployment-and-Maintenance/Deployment-Guide_timecho.md @@ -1163,4 +1163,314 @@ Run the remove-datanode script on an active DataNode: ### FAQ -See [FAQ](https://iotdb.apache.org/UserGuide/Master/FAQ/FAQ-for-cluster-setup.html). \ No newline at end of file +See [FAQ](https://iotdb.apache.org/UserGuide/Master/FAQ/FAQ-for-cluster-setup.html). + +## AINode deployment + +### Installation environment + +#### Recommended Operating System + +Ubuntu, CentOS, MacOS + +#### Runtime Environment + +AINode currently requires Python 3.8 or higher with pip and venv tools. + +For networked environments, AINode creates a virtual environment and downloads runtime dependencies automatically, no additional configuration is needed. + +In case of a non-networked environment, you can download it from https://cloud.tsinghua.edu.cn/d/4c1342f6c272439aa96c/to get the required dependencies and install them offline. + +### Installation steps + +Users can download the AINode software installation package, download and unzip it to complete the installation of AINode. You can also download the source code from the code repository and compile it to get the installation package. + +### Software directory structure + +After downloading and extracting the software package, you can get the following directory structure + +```Shell +|-- apache-iotdb-AINode-bin + |-- lib # package binary executable with environment dependencies + |-- conf # store configuration files + - iotdb-AINode.properties + |-- sbin # AINode related startup scripts + - start-AINode.sh + - start-AINode.bat + - stop-AINode.sh + - stop-AINode.bat + - remove-AINode.sh + - remove-AINode.bat + |-- licenses + - LICENSE + - NOTICE + - README.md + - README_ZH.md + - RELEASE_NOTES.md +``` + +- **lib:** AINode's compiled binary executable and related code dependencies. +- **conf:** contains AINode's configuration items, specifically the following configuration items +- **sbin:** AINode's runtime script, which can start, remove and stop AINode. + +### Start AINode + +After completing the deployment of Seed-ConfigNode, you can add an AINode node to support the model registration and inference functions. After specifying the information of IoTDB cluster in the configuration item, you can execute the corresponding commands to start AINode and join the IoTDB cluster. + +Note: Starting AINode requires that the system environment contains a Python interpreter of 3.8 or above as the default interpreter, so users should check whether the Python interpreter exists in the environment variables and can be directly invoked through the `python` command before using it. + +#### Direct Start + +After obtaining the installation package files, you can directly start AINode for the first time. + +The startup commands on Linux and MacOS are as follows: + +```Shell +> bash sbin/start-AINode.sh +``` + +The startup command on windows is as follows: + +```Shell +> sbin\start-AINode.bat +``` + +If start AINode for the first time and do not specify the path to the interpreter, the script will create a new venv virtual environment in the root directory of the program using the system Python interpreter, and install the third-party dependencies of AINode and the main program of AINode in this environment automatically and successively. **This process will generate a virtual environment of about 1GB in size, so please reserve space for installation**. On subsequent startups, if the path to the interpreter is not specified, the script will automatically look for the newly created venv environment above and start AINode without having to install the program and dependencies repeatedly. + +Note that it is possible to activate reinstall with -r if you wish to force a reinstall of AINode proper on a certain startup, this parameter will reinstall AINode based on the files under lib. + +Linux和MacOS： + +```Shell +> bash sbin/start-AINode.sh -r +``` + +Windows： + +```Shell +> sbin\start-AINode.bat -r +``` + +For example, a user replaces a newer version of the AINode installer in the lib, but the installer is not installed in the user's usual environment. In this case, you need to add the -r option at startup to instruct the script to force a reinstallation of the main AINode program in the virtual environment to update the version. + +#### Specify a customized virtual environment + +When starting AINode, you can specify a virtual environment interpreter path to install the AINode main program and its dependencies to a specific location. Specifically, you need to specify the value of the parameter ain_interpreter_dir. + +Linux and MacOS： + +```Shell +> bash sbin/start-AINode.sh -i xxx/bin/python +``` + +Windows： + +```Shell +> sbin\start-AINode.bat -i xxx\Scripts\python.exe +``` + +When specifying the Python interpreter please enter the address of the **executable file** of the Python interpreter in the virtual environment. Currently AINode **supports virtual environments such as venv, ****conda****, etc.** **Inputting the system Python interpreter as the installation location** is not supported. In order to ensure that scripts are recognized properly, please **use absolute paths whenever possible**! + +#### Join the cluster + +The AINode startup process automatically adds the new AINode to the IoTDB cluster. After starting the AINode you can verify that the node was joined successfully by entering the SQL for the cluster query in IoTDB's cli command line. +```Shell +IoTDB> show cluster ++------+----------+-------+---------------+------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710|UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730|UNKNOWN|190e303-dev| +| 2| AINode|Running| 127.0.0.1| 10810|UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------+-----------+ + +IoTDB> show cluster details ++------+----------+-------+---------------+------------+-------------------+----------+-------+-------+-------------------+-----------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|ConfigConsensusPort|RpcAddress|RpcPort|MppPort|SchemaConsensusPort|DataConsensusPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------------------+----------+-------+-------+-------------------+-----------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710| 10720| | | | | |UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730| | 0.0.0.0| 6667| 10740| 10750| 10760|UNKNOWN|190e303-dev| +| 2| AINode|Running| 127.0.0.1| 10810| | 0.0.0.0| 10810| | | |UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------------------+----------+-------+-------+-------------------+-----------------+-------+-----------+ + +IoTDB> show AINodes ++------+-------+----------+-------+ +|NodeID| Status|RpcAddress|RpcPort| ++------+-------+----------+-------+ +| 2|Running| 127.0.0.1| 10810| ++------+-------+----------+-------+ +``` + +### Remove AINode + +When it is necessary to move an already connected AINode out of the cluster, the corresponding removal script can be executed. + +The commands on Linux and MacOS are as follows: + +```Shell +> bash sbin/remove-AINode.sh +``` + +The startup command on windows is as follows: + +```Shell +> sbin/remove-AINode.bat +``` + +After removing the node, information about the node will not be available. + +```Shell +IoTDB> show cluster ++------+----------+-------+---------------+------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710|UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730|UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------+-----------+ +``` + +In addition, if the location of the AINode installation was previously customized, then the remove script should be called with the corresponding path as an argument: + +Linux and MacOS： + +```Shell +> bash sbin/remove-AINode.sh -i xxx/bin/python +``` + +Windows： + +```Shell +> sbin\remove-AINode.bat -i 1 xxx\Scripts\python.exe +``` + +Similarly, script parameters that are persistently modified in the env script will also take effect when the removal is performed. + +If a user loses a file in the data folder, AINode may not be able to remove itself locally, and requires the user to specify the node number, address and port number for removal, in which case we support the user to enter parameters for removal as follows + +Linux and MacOS： + +```Shell +> bash sbin/remove-AINode.sh -t /: +``` + +Windows： + +```Shell +> sbin\remove-AINode.bat -t /: +``` + +### Stop AINode + +If you need to stop a running AINode node, execute the appropriate shutdown script. + +The commands on Linux and MacOS are as follows: + +``` Shell. +> bash sbin/stop-AINode.sh +``` + +The startup command on windows is as follows: + +```Shell +> sbin/stop-AINode.bat +``` + +At this point the exact state of the node is not available and the corresponding management and reasoning functions cannot be used. If you need to restart the node, just execute the startup script again. + +```Shell +IoTDB> show cluster ++------+----------+-------+---------------+------------+-------+-----------+ +|NodeID| NodeType| Status|InternalAddress|InternalPort|Version| BuildInfo| ++------+----------+-------+---------------+------------+-------+-----------+ +| 0|ConfigNode|Running| 127.0.0.1| 10710|UNKNOWN|190e303-dev| +| 1| DataNode|Running| 127.0.0.1| 10730|UNKNOWN|190e303-dev| +| 2| AINode|UNKNOWN| 127.0.0.1| 10790|UNKNOWN|190e303-dev| ++------+----------+-------+---------------+------------+-------+-----------+ +``` + +### Script parameter details + +Two parameters are supported during AINode startup, and their specific roles are shown below: + +| **Name** | **Action Script** | Tag | **Description** | **Type** | **Default Value** | Input Method | +| ------------------- | ---------------- | ---- | ------------------------------------------------------------ | -------- | ---------------- | --------------------- | +| ain_interpreter_dir | start remove env | -i | The path to the interpreter of the virtual environment in which AINode is installed; absolute paths are required. | String | Read environment variables by default | Input on call + persistent modifications | +| ain_remove_target | remove stop | -t | AINode shutdown can specify the Node ID, address, and port number of the target AINode to be removed, in the format of `/:` | String | Null | Input on call | +| ain_force_reinstall | start remove env | -r | This script checks the version of the AINode installation, and if it does, it forces the installation of the whl package in lib if the version is not correct. | Bool | false | Input on call | +| ain_no_dependencies | start remove env | -n | Specifies whether to install dependencies when installing AINode, if so only the main AINode program will be installed without dependencies. | Bool | false | Input on call | + +Besides passing in the above parameters when executing the script as described above, it is also possible to modify some of the parameters persistently in the `AINode-env.sh` and `AINode-env.bat` scripts in the `conf` folder. + +`AINode-env.sh`： + +```Bash +# The defaulte venv environment is used if ain_interpreter_dir is not set. Please use absolute path without quotation mark +# ain_interpreter_dir= +``` + +`AINode-env.bat`： + +```Plain +@REM The defaulte venv environment is used if ain_interpreter_dir is not set. Please use absolute path without quotation mark +@REM set ain_interpreter_dir= +``` + +Uncomment the corresponding line after writing the parameter value and save it to take effect the next time you execute the script. + +### AINode configuration items + +AINode supports modifying some necessary parameters. The following parameters can be found in the `conf/iotdb-AINode.properties` file and modified for persistence: + +| **Name** | **Description** | **Type** | **Default Value** | **Modified Mode of Effect** | +| --------------------------- | ------------------------------------------------------------ | -------- | ------------------ | ---------------------------- | +| ain_target_config_node_list | ConfigNode address registered at AINode startup | String | 10710 | Only allow to modify before the first startup | +| ain_inference_rpc_address | Addresses where AINode provides services and communications | String | 127.0.0.1 | Effective after reboot | +| ain_inference_rpc_port | AINode provides services and communication ports | String | 10810 | Effective after reboot | +| ain_system_dir | AINode metadata storage path, the starting directory of the relative path is related to the operating system, it is recommended to use the absolute path. | String | data/AINode/system | Effective after reboot | +| ain_models_dir | AINode stores the path to the model file. The starting directory of the relative path is related to the operating system, and an absolute path is recommended. | String | data/AINode/models | Effective after reboot | +| ain_logs_dir | The path where AINode stores the logs. The starting directory of the relative path is related to the operating system, and it is recommended to use the absolute path. | String | logs/AINode | Effective after reboot | + +### Frequently Asked Questions + +1. **Not found venv module error when starting AINode** + +When starting AINode using the default method, a python virtual environment is created in the installation package directory and dependencies are installed, thus requiring the installation of the venv module. Generally speaking, python 3.8 and above will come with venv, but for some systems that come with python environment may not fulfill this requirement. There are two solutions when this error occurs (either one or the other): + +- Install venv module locally, take ubuntu as an example, you can run the following command to install the venv module that comes with python. Or install a version of python that comes with venv from the python website. + +```SQL +apt-get install python3.8-venv +``` + +- Specify the path to an existing python interpreter as the AINode runtime environment via -i when running the startup script, so that you no longer need to create a new virtual environment. + +2. **Compiling the python environment in CentOS7** + +The new environment in centos7 (comes with python3.6) does not meet the requirements to start mlnode, you need to compile python3.8+ by yourself (python is not provided as a binary package in centos7) + +- Install OpenSSL + +> Currently Python versions 3.6 to 3.9 are compatible with OpenSSL 1.0.2, 1.1.0, and 1.1.1. + +Python requires that we have OpenSSL installed on our system, which can be found at https://stackoverflow.com/questions/56552390/how-to-fix-ssl-module-in-python-is-not-available-in-centos + +- Installation and compilation of python + +Download the installation package from the official website and extract it using the following specifications + +```SQL +wget https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz +tar -zxvf Python-3.8.1.tgz +``` + +Compile and install the corresponding python packages. + +```SQL +./configure prefix=/usr/local/python3 -with-openssl=/usr/local/openssl +make && make install +``` + +1. **Windows compilation problem like "error: Microsoft Visual** **C++** **14.0 or greater is required..." compilation problem** on windows. + +The corresponding error is usually caused by an insufficient version of c++ or setuptools, you can find the appropriate solution at https://stackoverflow.com/questions/44951456/pip-error-microsoft-visual-c-14-0-is-required +you can find a suitable solution there. \ No newline at end of file