云鉴权>> huststore>> 返回
项目作者： Qihoo360

项目描述 ：
High-performance Distributed Storage
高级语言： C
项目地址: git://github.com/Qihoo360/huststore.git
创建时间： 2016-04-27T12:16:12Z
项目社区：https://github.com/Qihoo360/huststore
开源协议：GNU Lesser General Public License v3.0
下载


简体中文

huststore - High-performance Distributed Storage

huststore logo

huststore is a open source high performance distributed database system. It not only provides key-value storage service with extremely high performance, up to hundreds of thousands QPS, but also supports data structures like hash, set, sorted set, etc. Also, it can store binary data as value from a key-value pair, and thus can be used as an alternative of Redis.

In addtion, huststore implements a distributed message queue by integrating a special HA module, features including message Push Stream, and message Publish-SubScribe.

Features

huststore has two core components, hustdb and HA. hustdb is a database engine developed by our own, in the fundamental architecture. HA is implemented as a nginx module. It is well-known that nginx is a industry-proven high quality code base, thus by inheriting it huststore gains the below advantages:

  • High Throughput
    hustdb uses libevhtp, a open source network library, as the inner network communication system, by incorporating it with high-performance storage engine, hustdb achieves a extremely high performance, the benchmark shows that QPS hits hundreds of thousands and even more.

  • High Concurrency
    Please refer to concurrency report of nginx for more details.

  • High Availability
    huststore architecture provides Replication (master-master) and load balance support. Therefore, the availability of HA is guaranteed by master-worker design. When one of worker process is down, the master will load another workder process, since multiple workers work independently, the HA is guaranteed to work steadily.
    The fundamental design architecture of huststore guarantees the high availability, by using master-master architecture, when one of the storage node fails, HA module will re-direct the request to another living master node. Also, when a node failure happens, HA cluster will automatically re-balance the data distribution, thus avoid single point of failure.
    In addition, HA cluster uses a distributed architecture design by incorporating LVS as the director, each HA node is separated and work independently. When one of the HA node is down, LVS will re-direct the request to other available HA node, thus avoids HA‘s failure on single point node.

  • Language-free Interface
    huststore use http as the communication protocol, therefore the client side implementation is not limited in any specific programming language.

  • Persistence
    You do not need to worry about the loss of data as most of interfaces will persist data to disk.

  • Support Binary Key-Value

  • Support Version Clock

Operation and Maintenance

Architect

architect

Deployment

  • Distributed KV storage : HA (hustdb ha) + DB (hustdb)
  • Distributed Message Queue : HA (hustmq ha) + DB (hustdb)

Database Engine

hustdb

Dependency

Platforms

Tested platforms so far:

Platform Description
CentOS 6.x & 7.x kernel >= 2.6.32 (GCC 4.4.7)

Quick Start

Read the Quick Start.

Documents

Above includes detailed documents of design, deployments, API usage and test samples. You can refer quickly to common problems in FAQ part.

Performance

Environment

  1. CPU: Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz (2socket*16cores)
  2. Memory: 192G
  3. Disk: Intel SSD DC S3520 Series (800GB, 2.5in SATA 6Gb/s, 3D1, MLC)
  4. Network Adapter: Intel Ethernet 10G 2P X520 Adapter
  5. OS: CentOS Linux release 7.2.1511 (3.10.0-327.el7.x86_64)

Test 1 - Worker

Purpose

With different number of hustdb worker threads, we test hustdb’s max QPS.

Condition

storage capacity : 512GB

concurrent connection : 1000

value : 1KB

md5db cache : disabled

data compression : disabled

Threads are not bound to CPU cores.

Result

Description : Horizontal axis is the number of worker threads; Vertical axis is the QPS. The value size is 1KB .

benchmark_workers

Conclusion

The best hustdb worker threads number is 36-40.

Test 2 - RTT

Purpose

With the optimal worker threads number, we test hustdb’s round-trip time.

Condition

storage capacity : 512GB

concurrent connection : 200

value : 1KB

md5db cache : disabled

data compression : disabled

Threads are not bound to CPU cores.

Result

  1. # GET
  2. Thread Stats   Avg      Stdev     Max   +/- Stdev
  3.     Latency   238.94us   95.42us   9.33ms   81.03%
  4.     Req/Sec    31.44k     1.09k   36.30k    63.68%
  5. Latency Distribution
  6.     50%  220.00us
  7.     75%  288.00us
  8.     90%  343.00us
  9.     99%  482.00us
  10. 11333230 requests in 15.10s, 11.10GB read
  11. Requests/sec: 750635.81
  12. Transfer/sec:    752.65MB
  13. --------------------------------------------------
  14. [Latency Distribution]  0.01%  0.07ms
  15. [Latency Distribution]   0.1%  0.07ms
  16. [Latency Distribution]   0.5%  0.09ms
  17. [Latency Distribution]     1%  0.09ms
  18. [Latency Distribution]     3%  0.12ms
  19. [Latency Distribution]     5%  0.13ms
  20. [Latency Distribution]    10%  0.15ms
  21. [Latency Distribution]    20%  0.17ms
  22. [Latency Distribution]    30%  0.19ms
  23. [Latency Distribution]    40%  0.20ms
  24. [Latency Distribution]    50%  0.22ms
  25. [Latency Distribution]    60%  0.25ms
  26. [Latency Distribution]    70%  0.28ms
  27. [Latency Distribution]    80%  0.30ms
  28. [Latency Distribution]    90%  0.34ms
  29. [Latency Distribution]    91%  0.35ms
  30. [Latency Distribution]    92%  0.36ms
  31. [Latency Distribution]    93%  0.37ms
  32. [Latency Distribution]  93.5%  0.37ms
  33. [Latency Distribution]    94%  0.38ms
  34. [Latency Distribution]  94.5%  0.38ms
  35. [Latency Distribution]    95%  0.39ms
  36. [Latency Distribution]  95.5%  0.39ms
  37. [Latency Distribution]    96%  0.40ms
  38. [Latency Distribution]  96.5%  0.40ms
  39. [Latency Distribution]    97%  0.41ms
  40. [Latency Distribution]  97.5%  0.42ms
  41. [Latency Distribution]    98%  0.43ms
  42. [Latency Distribution]  98.5%  0.45ms
  43. [Latency Distribution]    99%  0.48ms
  44. [Latency Distribution]  99.1%  0.49ms
  45. [Latency Distribution]  99.2%  0.50ms
  46. [Latency Distribution]  99.3%  0.51ms
  47. [Latency Distribution]  99.4%  0.52ms
  48. [Latency Distribution]  99.5%  0.53ms
  49. [Latency Distribution]  99.6%  0.56ms
  50. [Latency Distribution]  99.7%  0.59ms
  51. [Latency Distribution]  99.8%  0.64ms
  52. [Latency Distribution]  99.9%  0.76ms
  53. [Latency Distribution]  99.99%  1.85ms
  54. [Latency Distribution]  99.999%  4.07ms
  56. # PUT
  57. Thread Stats   Avg      Stdev     Max   +/- Stdev
  58.     Latency   495.13us  393.71us  21.29ms   93.06%
  59.     Req/Sec    16.37k     1.33k   23.72k    74.26%
  60. Latency Distribution
  61.     50%  447.00us
  62.     75%  623.00us
  63.     90%  815.00us
  64.     99%    1.28ms
  65. 17628712 requests in 45.10s, 1.53GB read
  66. Requests/sec: 390880.11
  67. Transfer/sec:     34.67MB
  68. --------------------------------------------------
  69. [Latency Distribution]  0.01%  0.09ms
  70. [Latency Distribution]   0.1%  0.10ms
  71. [Latency Distribution]   0.5%  0.12ms
  72. [Latency Distribution]     1%  0.12ms
  73. [Latency Distribution]     3%  0.14ms
  74. [Latency Distribution]     5%  0.17ms
  75. [Latency Distribution]    10%  0.20ms
  76. [Latency Distribution]    20%  0.26ms
  77. [Latency Distribution]    30%  0.33ms
  78. [Latency Distribution]    40%  0.39ms
  79. [Latency Distribution]    50%  0.45ms
  80. [Latency Distribution]    60%  0.51ms
  81. [Latency Distribution]    70%  0.58ms
  82. [Latency Distribution]    80%  0.67ms
  83. [Latency Distribution]    90%  0.81ms
  84. [Latency Distribution]    91%  0.84ms
  85. [Latency Distribution]    92%  0.86ms
  86. [Latency Distribution]    93%  0.89ms
  87. [Latency Distribution]  93.5%  0.90ms
  88. [Latency Distribution]    94%  0.92ms
  89. [Latency Distribution]  94.5%  0.93ms
  90. [Latency Distribution]    95%  0.95ms
  91. [Latency Distribution]  95.5%  0.97ms
  92. [Latency Distribution]    96%  0.99ms
  93. [Latency Distribution]  96.5%  1.02ms
  94. [Latency Distribution]    97%  1.05ms
  95. [Latency Distribution]  97.5%  1.08ms
  96. [Latency Distribution]    98%  1.13ms
  97. [Latency Distribution]  98.5%  1.19ms
  98. [Latency Distribution]    99%  1.28ms
  99. [Latency Distribution]  99.1%  1.30ms
  100. [Latency Distribution]  99.2%  1.33ms
  101. [Latency Distribution]  99.3%  1.37ms
  102. [Latency Distribution]  99.4%  1.41ms
  103. [Latency Distribution]  99.5%  1.47ms
  104. [Latency Distribution]  99.6%  1.56ms
  105. [Latency Distribution]  99.7%  1.73ms
  106. [Latency Distribution]  99.8%  2.24ms
  107. [Latency Distribution]  99.9%  4.23ms
  108. [Latency Distribution]  99.99%  7.22ms
  109. [Latency Distribution]  99.999%  9.62ms

Test 3 - vs Redis

Version

Tools

Condition

storage capacity : 512GB

md5db cache : disabled

data compression : disabled

Threads are not bound to CPU cores.

Arguments

abbr concurrent connection value
C1000-512B 1000 512B
C1000-1K 1000 1KB
C1000-4K 1000 4KB
C2000-512B 2000 512B
C2000-1K 2000 1KB
C2000-4K 2000 4KB

PUT

benchmark_put

GET

benchmark_get

See more details in here

LICENSE

huststore is licensed under LGPL-3.0, a very flexible license to use.

Authors

More

  • Nginx module development kit - hustngx