Readme.en.md

NewLife.Redis - Redis Client Component

NewLife.Redis is a Redis client component designed for high-performance real-time big data processing.
The Redis protocol is implemented in the Redis/RedisClient located in the X component, and this library extends it with advanced features such as list structures, hash structures, and queues.

Source: https://github.com/NewLifeX/NewLife.Redis
Nuget: NewLife.Redis
Tutorial: https://newlifex.com/core/redis

---

Features

• Widely used for real-time big data processing at ZTO since 2017. More than 200 Redis instances have been running stably for over a year, processing nearly 100 million package data entries daily, with 8 billion calls per day.
• Low latency, with Get/Set operations averaging 200-600μs (including round-trip network communication).
• High throughput with an in-built connection pool, supporting up to 100,000 concurrent connections.
• High performance, supports binary serialization.

---

Redis Experience Sharing

• Deploy multiple instances on Linux, with the number of instances equal to the number of processors. Each instance's maximum memory equals the physical memory of the machine to avoid memory overflow in a single instance.
• Store massive data (over 1 billion entries) across multiple instances by hashing keys (Crc16/Crc32), greatly improving read and write performance.
• Use binary serialization instead of common JSON serialization.
• Design the size of each key-value pair carefully, including but not limited to batch retrievals. The goal is to keep each network packet around 1.4k bytes to reduce communication overhead.
• Redis Get/Set operations average 200-600μs (including round-trip network communication). Use this as a benchmark to assess the network environment and Redis client components.
• Use pipelining to combine a batch of commands.
• Redis’s main performance bottlenecks are serialization, network bandwidth, and memory size. Excessive use can also cause processor bottlenecks.
• Other optimizations can be explored.

The above experiences come from over a year of stable operation with over 300 instances and more than 4TB of space, ordered by importance and should be applied as needed based on the scenario.

---

Recommended Usage

It is recommended to use the Singleton pattern. Redis internally uses a connection pool and supports multi-threaded concurrent access.

public static class RedisHelper
{
    /// <summary>
    /// Redis instance
    /// </summary>
    public static FullRedis redisConnection { get; set; } = new FullRedis("127.0.0.1:6379", "123456", 4);
}

Console.WriteLine(RedisHelper.redisConnection.Keys);

---

Basic Redis

The Redis implementation follows the standard protocol and basic string operations, with the complete implementation provided by the independent open-source project NewLife.Redis.
It uses a connection pool and synchronous blocking architecture, offering ultra-low latency (200-600μs) and extremely high throughput.
It is widely used in real-time big data processing in the logistics industry and has been validated with daily call volumes of 10 billion.

// Instantiate Redis, default port 6379 can be omitted, two ways to write the password
//var rds = new FullRedis("127.0.0.1", null, 7);
var rds = new FullRedis("127.0.0.1:6379", "pass", 7);
//var rds = new FullRedis();
//rds.Init("server=127.0.0.1:6379;password=pass;db=7");
rds.Log = XTrace.Log;

Basic Operations

Before performing basic operations, we need to do some preparation:

• Create a new console project and add XTrace.UseConsole(); at the beginning of the main function to easily view debug logs.
• Before testing specific code, ensure you have instantiated either MemoryCache or Redis.
• Prepare a model class User:

class User
{
    public String Name { get; set; }
    public DateTime CreateTime { get; set; }
}

Add, Delete, Update, Query:

var rds = new FullRedis("127.0.0.1", null, 7);
rds.Log = XTrace.Log;
rds.ClientLog = XTrace.Log; // Debug log. Comment out for production use.
var user = new User { Name = "NewLife", CreateTime = DateTime.Now };
rds.Set("user", user, 3600);
var user2 = rds.Get<User>("user");
XTrace.WriteLine("Json: {0}", user2.ToJson());
XTrace.WriteLine("Json: {0}", rds.Get<String>("user"));
if (rds.ContainsKey("user")) XTrace.WriteLine("Exists!");
rds.Remove("user");

Execution Result:

14:14:25.990  1 N - SELECT 7
14:14:25.992  1 N - => OK
14:14:26.008  1 N - SETEX user 3600 [53]
14:14:26.021  1 N - => OK
14:14:26.042  1 N - GET user
14:14:26.048  1 N - => [53]
14:14:26.064  1 N - GET user
14:14:26.065  1 N - => [53]
14:14:26.066  1 N - Json: {"Name":"NewLife","CreateTime":"2018-09-25 14:14:25"}
14:14:26.067  1 N - EXISTS user
14:14:26.068  1 N - => 1
14:14:26.068  1 N - Exists!
14:14:26.069  1 N - DEL user
14:14:26.070  1 N - => 1

When saving complex objects, the default serialization method is JSON. Therefore, when retrieving the result as a string, you'll find it is in JSON format.
Redis strings are essentially binary data with length prefixes, where [53] indicates 53 bytes of binary data.

Collection Operations

GetAll/SetAll are commonly used batch operations in Redis, allowing you to get or set multiple keys simultaneously, often achieving more than 10x throughput.

Batch operations:

var rds = new FullRedis("127.0.0.1", null, 7);
rds.Log = XTrace.Log;
rds.ClientLog = XTrace.Log; // Debug log. Comment out for production use.
var dic = new Dictionary<String, Object>
{
    ["name"] = "NewLife",
    ["time"] = DateTime.Now,
    ["count"] = 1234
};
rds.SetAll(dic, 120);

var vs = rds.GetAll<String>(dic.Keys);
XTrace.WriteLine(vs.Join(",", e => $"{e.Key}={e.Value}"));

Execution Result:

MSET name NewLife time 2018-09-25 15:56:26 count 1234
=> OK
EXPIRE name 120
EXPIRE time 120
EXPIRE count 120
MGET name time count
name=NewLife,time=2018-09-25 15:56:26,count=1234

In collection operations, there are also GetList/GetDictionary/GetQueue/GetSet for various types of collections like Redis lists, hashes, queues, and sets.
The basic Redis version does not support these collections, but the full version NewLife.Redis does, while MemoryCache directly supports them.

Advanced Operations

• Add: Adds a key if it does not exist, returns false if the key already exists.
• Replace: Replaces an existing value with a new value and returns the old value.
• Increment: Increments a value atomically.
• Decrement: Decrements a value atomically.

Advanced operations:

var rds = new FullRedis("127.0.0.1", null, 7);
rds.Log = XTrace.Log;
rds.ClientLog = XTrace.Log; // Debug log. Comment out for production use.
var flag = rds.Add("count", 5678);
XTrace.WriteLine(flag ? "Add Success" : "Add Failed");
var ori = rds.Replace("count", 777);
var count = rds.Get<Int32>("count");
XTrace.WriteLine("count changed from {0} to {1}", ori, count);

rds.Increment("count", 11);
var count2 = rds.Decrement("count", 10);
XTrace.WriteLine("count={0}", count2);

Execution Result:

SETNX count 5678
=> 0
Add Failed
GETSET count 777
=> 1234
GET count
=> 777
count changed from 1234 to 777
INCRBY count 11
=> 788
DECRBY count 10
=> 778
count=778

---

Performance Testing

The Bench tool will perform pressure testing by dividing the workload into multiple groups based on the number of threads.
Parameters:

• rand: Whether to randomly generate keys/values.
• batch: Batch size, which optimizes read/write operations using GetAll/SetAll.

By default, Redis sets AutoPipeline=100, meaning it automatically pipelines operations for better performance during read and write operations when there’s no batching.

---

Redis's Siblings

Redis implements the ICache interface, with its sibling, MemoryCache, being an in-memory cache with a throughput in the tens of millions.
It is highly recommended to design applications using the ICache interface. Use MemoryCache for small data, and when the data grows (to 100,000 entries), switch to Redis without needing to modify the business logic code.

Redis Server Support

1. Version 3.2 and above:
   From version 3.2 onwards, FullRedis supports all Redis versions.

2. Stream Data Type:
   FullRedis supports the Stream data type, which was introduced in Redis 5.0, allowing the storage and manipulation of message streams.

3. LUA Not Supported:
   FullRedis does not support LUA scripting, meaning you cannot execute Lua scripts through this client.