Building streaming data server

Mar 31, 2012 at 9:45 PM
Edited Mar 31, 2012 at 9:46 PM

Hi, I am working on a server for the timeseries database, and would like to use your project instead of dealing with the lower level plumbing, etc. Could you recommend a starting point.

My first goal is to implement a simple request response socket server that would provide functionality similar to IEnumerable<T>

  • CLIENT sends request GetData(params)
    • params is a custom binary encoded object with generic values that the client and the server will both know about at the runtime
  • SERVER Starts sending a stream of binary data without knowing itself when it will end (could be anywhere from a few bytes to terabytes). The client at any moment could stop iteration (by calling Dispose() on the returned data stream), or the server may finish sending the data.

It might be good to be able to reuse connections, but I don't know how it is possible if both the server and the client can abruptly stop the data flow.

Client usage scenario - get all IBM trades for the past 5 days until now, and keep giving real-time price updates

foreach(var trade in 
   server.GetData<DateTime, Trade>("IBM Stock Trades", DateTime.Now.AddDays(-5)))
{
    // process trades
    if (...)
       break; // stop at any moment 
}

 

Coordinator
Apr 1, 2012 at 2:42 AM

If you prefer to define your own binary protocol, you need to use CustomProtocol. There are sample code in QuickStart of source code.

But if you want to use an existing protocol, I suggest WebSocket to you. You can use SuperWebSocket which is a websocket server powered by SuperSocket. And you need't pay more efforts to client, because you can use any WebSocket client (I recommend WebSocket4Net) to communicate with server.

 

The data processing should be async if you want a high performance client.

The client should work like that:

client.On("NewTrade", HandleNewTradeData)

void HandleNewTradeData(TradeData data) {

   //data processing code

}

Coordinator
Apr 5, 2012 at 4:01 AM

I suggest you to use SuperWebSocket, then you needn't take care of underlying binary protocol and you can use any websocket client which you like.