The ETFs of Hong Kong Digital Assets made a strong debut on April 15th, injecting powerful momentum into the development of the digital asset market and also providing new investment opportunities for investors. As an investment product, digital assets are sweeping the globe in an unstoppable trend.

In the past month, mainstream digital assets such as BTC and ETH have experienced huge fluctuations, announcing the start of a new round of bull market for virtual currencies. This has not only attracted the attention of a large number of investors, but also posed greater challenges to the technology of digital currency trading platforms.

Challenges of Data Storage and Processing

The digital currency trading market differs significantly from traditional financial markets. It operates on a 24/7 basis, generating a total of more than 10TB of market data per day, and this volume continues to grow. Moreover, there exists an extreme imbalance in the amount of market data among different cryptocurrencies, with the majority concentrated in the top assets.

Furthermore, the depth of market data varies greatly across currencies, with some having only a dozen levels while others exceeding a thousand. The market fluctuations in digital currencies are exceptionally rapid and volatile, with significant price swings occurring frequently. This poses stringent requirements on system latency. Any slight delay may lead to trading failures, causing significant losses to investors.

Breaking Through with Time Series Databases

Faced with these challenges, time series databases have become the ideal solution for data storage and processing in financial trading platforms and quantitative investment funds. Time series databases are specially designed for processing time series data and can efficiently store and query massive amounts of data. Their advantages lie in the ability to quickly process a large number of data write and query requests, meeting the real-time data needs of the digital currency trading market.

Time series databases effectively compress time series data thus reducing storage costs. They can efficiently query historical data to support various complex time series analyses. With these advantages, they have been widely applied by traditional financial enterprises and organizations such as brokerages, banks, mutual / hedgefunds, etc., laying a solid foundation for safe and stable operation of platforms.

In terms of application scenarios, financial institutions can use technical analysis methods to predict market price trends through calculation of indicators and visualization charts, helping teams make trading decisions. Since technical analysis only studies the price and does not focus on the type of assets, this method is applicable to all trading markets. After the formation of the cryptocurrency trading market, technical analysis has become an important part of cryptocurrency trading.

The following section will show you how to implement 8 commonly used technical indicators through high-performance real-time computing, and combine them through visualization technology to build a trading dashboard for digital asset market. Through these visualization dashboards, we can identify market trends, observe price fluctuations, explore market structure, and thus provide more comprehensive insights for decision-making.

For this demonstration, we will use DolphinDB for real-time computing of the indicators. DolphinDB is a real-time platform for analytics and stream processing powered by high-performance time series database. It is characterized by its lightweight, one-stop, and powerful computing performance. Its scalable visualization capabilities help users easily build an interactive dashboard.

Currently, DolphinDB has provided data processing services for more than a hundred institutions in the traditional financial field, including Guotai Junan Securities, Fullgoal Fund, China Merchants Bank, etc. In the digital asset field, it has also served exchanges, market makers, and other institutions.

8 Commonly Used Technical Indicators

According to public data, there are more than 100 technical indicators on the market. We will select the 8 most commonly used technical indicators including Moving Average, K-line, Awesome Oscillator, Bollinger Band, etc.

Moving Average (MA)

The moving average price, also known as the moving average line, is a curve generated by calculating the average price over a certain period, commonly used to identify trend reversal points, support levels, and resistance levels. DolphinDB provides more than 1,500 built-in functions for financial trading. Here we use the built-in tm-functions. The following script can quickly calculate the moving average price indicator.

select trade Time, code as pair, price , tmavg(datetime(tradeTime),price,10s) as movingAvg10Sec , tmavg(datetime(tradeTime),price,30s) as movingAvg30Sec , tmavg(datetime(tradeTime),price,45s) as movingAvg45Sec from (select * from aggTradeStream10 where code =`BTCUSDT order by id asc) where tradeTime > temporalAdd(now(),-485m)

After creating a panel in Grafana (for instructions on how to connect DolphinDB with Grafana, please refer to the end of this chapter), select the visualization type as Time series and Graph styles option as Line, input the above statement and execute it, you will get the output as shown in the figure below: The panel “Moving Average of BTC-USDT“ charts the real-time price of BTC-USDT’s aggregated trades over the past 5 minutes, and average prices calculated over the 10-second, 30-second, and 45-second sliding windows.

K-line Chart

K-line is one of the most important technical indicators. We can implement the real-time calculation of OHLC prices with the following script.

select first(price) as open, last(price) as close, min(price) as low, max(price) as high,sum(quantity) as volumefrom aggTradeStream10where temporalAdd(now(),-540,'m') < tradeTime and code=`BTCUSDTgroup by bar(tradeTime,1s)

In Grafana’s panel (for instructions on how to connect DolphinDB with Grafana, please refer to Chapter 3), select the visualization type as Candlestick in the visualization list.

The 1-second OHLC bars of the previous 24 hours are derived from the dataset. The panel “OHLC Bars of BTC-USDT“ below displays 1-second OHLC bars for the BTC-USDT over last 5 minutes.

Relative Strength Index（RSI）

The Relative Strength Index (RSI) is a momentum based oscillator to chart the current and historical strength or weakness of a stock or market based on the closing prices of a recent trading period. This indicator can effectively identify overbought and oversold trends in the trading market and is one of the most popular oscillator indicators. Select the visualization type as Time series in the visualization list and choose Line in the Graph styles option.

use taselect time, rsi(close,20) as RSI, 70 as upperbond, 30 as lowerbond from (select first(price)as open, last(price) as close, min(price) as lo, max(price) as hi, sum(quantity) as volfrom aggTradeStream10where temporalAdd(now(),-32,'H') <= tradeTime and code="BTCUSDT"group by code, bar(tradeTime,1s) as time)

The query above calculates the RSI based on the 1-second OHLC bars from the previous 24-hour period. The panel “RSI of BTC-USDT“ below displays the RSI line for BTC-USDT over last 5 minutes.

Moving Average Convergence Divergence（MACD）

Moving average convergence/divergence (MACD) is a trend-following momentum indicator that shows the relationship between two EMAs of market price. It helps investors identify price trends, measure trend momentum, and identify market entry points for buying or selling. It is a trend-following indicator that works well in volatile markets. Select the visualization type as Time series in the visualization list and choose Line in the Graph styles option.

use taselect time, macd(close) as `DIF`DEA`MACD, 0 as zeroline from (select first(price) as open,last(price) as close, min(price) as lo, max(price) as hi, sum(quantity) as volfrom aggTradeStream10where temporalAdd(now(),-32,'H') <= tradeTime and code="BTCUSDT"group by code, bar(tradeTime,1s) as time)

The query above calculates the MACD based on the 1-second OHLC bars from the previous 24-hour period.

The panel “MACD“ below displays the DIF line, DEA line, and MACD line for BTC-USDT trading pair over last 5 minutes.

Bollinger Bands

Bollinger Bands is a technical analysis tool defined by a set of trendlines, which are plotted as two standard deviations, both positively and negatively, away from a SMA of the price. It helps investors to identify when an asset is oversold or overbought. It is commonly used to analyze market volatility, confirm trend directions, and identify buy and sell signals. Select the visualization type as Time series in the visualization list and choose Line in the Graph styles option.

use taselect time, bBands(close,5,2,2,2) as `Low`Mid`High from (select first(price) as open,last(price) as close, min(price) as lo, max(price) as hi, sum(quantity) as volfrom aggTradeStream10where temporalAdd(now(),-32,'H') <= tradeTime and code="BTCUSDT"group by code, bar(tradeTime,1s) as time)

The query above calculates the Bollinger Bands based on the 1-second OHLC bars from the previous 24-hour period.

The panel “Bollinger Bands“ below displays the band of three lines (Middle, Upper, and Lower) for BTC-USDT trading pair over last 5 minutes.

Correlation

Select the visualization type as Time series in the visualization list and choose Line in the Graph styles option.

a = select avg(price) as price from aggTradeStream10 where code = `BTCUSDT or code =`ETHUSDT group by bar(tradetime,1s) as time,codeb = select corr(BTCUSDT, ETHUSDT) as corrVal from (select price from a pivot bytime,code) group by bar(time,1m) as timeselect time, tmavg(time, corrVal,1H) as corr1h, tmavg(time, corrVal,24H) as corr24h from b

The query first calculates the average prices for the BTC-USDT and ETH-USDT in 1-second intervals. Then, it computes the correlation between these two pairs over 1-minute intervals based on the moving average prices.

The panel “Correlation of BTC-USDT and ETH-USDT“ below shows the average correlation between the BTC-USDT and ETH-USDT trading pairs calculated over the 1-hour and 24-hour sliding windows.

Real-time Trades

Real-time trading data provides crucial insights into the current market dynamics of a trading pair. This information reveals buyer and seller sentiment, as well as the balance of power between long and short positions. Select the visualization type as Table in the visualization list.

select tradeTime as timestamp, code as pair, case when marketMaker = true then -1 *quantity else quantity end as quantity,case when marketMaker = true then round(-1*quantity*price,2) else round(quantity*price,2)end as consideration,case when (temporalAdd(now(), -8H)-tradeTime)\1000000<0.3 then "x" else "" end as newfrom aggTradeStream10 where tradeTime > temporalAdd(now(), -1, "M") order by tradeTimedesc limit 50

The panel “Real-time Trades“ below displays the last 50 real-time trades across 10 crypto trading pairs, with the data being updated every 5 seconds.

Real-time Trading Volume

The real-time trading volume indicator can illustrate the activity level and capital scale of the trading market, helping us to understand market sentiment. Select the visualization type as Time series in the visualization list and choose Points in the Graph styles option.

defg getA(quantity, price, marketMaker){a = iif(marketMaker==true, -1,1)return (a*quantity*price)[0]}select valfrom (select getA(quantity, price, marketMaker) as val from aggTradeStream10 wheretradeTime between startTime0 and endTime0 group by datetime(tradetime) astradetime,code) pivot by tradeTime, code

The panel “Real-time Trading Volume“ below depicts the real-time trading volume over the past 5 minutes for 10 crypto trading pairs. Positive values indicate buy orders, and negative values represent sell orders.

For those interested in the data mentioned above, you can try connecting Grafana to DolphinDB. Follow the tutorial to install the dolphindb-datasource plugin. Once the installation is successful and Grafana is restarted, search for "DolphinDB" in the DataSource settings to create a corresponding data source.

Fill in the corresponding IP address and port, along with the username and password that have access permissions to the required databases and tables, to create the data source. Click on “Save & Test” below. A successful connection will be indicated by a confirmation message, as shown in the image below.

After creating the data source, you can then visualize data in panels by building a dashboard.

In the dialog box that opens, select the saved DolphinDB data source. The Edit panel view will then open. We can now write DolphinDB query in the Query editor. The results will be displayed in the panel.

Building on our earlier exploration of technical indicator calculations, we now turn our attention to specific use cases that showcase the computational capabilities of time series databases. To illustrate these capabilities, we’ll look at examples from traditional finance, using DolphinDB as our benchmark. By examining the performance in conventional financial markets, we can infer the potential power and applications in cryptocurrency trading.

Configuration: CPU with 16 cores, 512GB of memory, and 4 SSD

• SQL Query and Aggregation Performance: Within a 27 billion-row dataset, queries and aggregations (including decompression) are completed in milliseconds.

• Pairwise Correlation Calculation: From a 27 billion-row dataset, the 500 most actively traded stocks (approximately 200 million data points) are selected for pairwise correlation calculation, which is completed in seconds.

• Trading Cost Calculation: Asof join and window join are performed on the trading table (27 million rows) and the quote table (87 million rows), completing in microseconds.

• Factor Calculation: The calculation of the trading alpha 98 (of the WorldQuant), using daily data from the full market over a one-year period, is completed in milliseconds.

• Data Downsampling: Within a 27 billion-row dataset, 6.5 billion high-frequency tick data points (300GB) are compressed to minute-level frequency (61 million), and the process is completed in 30 seconds.

• ArrayVector Structure: It can support efficient storage of multi-level market data with a compression ratio of 10:1.

• Real-time K-Line Synthesis and Factor Real-time Calculation: Replay of one day’s trading data across the full market (200 million rows) with real-time calculation of OHLC and dual moving average factors.

• FX Swap Valuation: Valuation calculation for 1 million FX swap contracts can be completed in 400 milliseconds.

• Distributed Linear Regression: Linear regression calculation for 1 billion data points can be completed in seconds.

• IOPV Calculation: Calculation of the IOPV for an ETF across all trade tick data points in a day can be completed in microseconds using a single core.

The above examples fully demonstrate the powerful capabilities of time-series databases in handling massive data processing, complex indicator calculations, multi-table join queries, real-time data analysis, financial derivative valuation, distributed computing, and high-concurrency calculations. They empower users with the ability to gain real-time insights into data, quickly identify patterns, and assist in decision-making. Time-series databases have become an essential component of the new generation of data infrastructure and are poised to lead the future development of data analysis technology.

Summary

As ETFs gain approval from the SEC and the Hong Kong government, digital assets are bound to enter the “institutional era.” Time series databases, with their high throughput, efficient querying, and excellent scalability, will record every transaction, event, and decision, thereby constructing a complete digital asset lifecycle.

By analyzing historical data, traders can effortlessly gain insights into market trends, forecast future movements, and develop the most timely trading strategies. This provides strong data support for the investment, trading, and management of digital assets.