WebSocket protocol is a new protocol developed to solve the problems of statelessness, short connection (usually) of http protocol and the server's inability to actively push data to the client. Its communication foundation is also based on TCP. Since older browsers may not support the WebSocket protocol, the two communication parties using the WebSocket protocol need to perform an extra handshake after three TCP handshakes. This time, the message format of the two communication parties is modified based on the HTTP protocol.
WebSocket handshake process
We will not repeat the process of TCP three times handshake here. Any network communication book has a detailed introduction. Let's introduce the last handshake process of WebSocket communication.
After the handshake starts, one party sends a message in http protocol format to the other party. The message format is roughly as follows:
one GET /realtime HTTP/1.1\r\n
two Host: 127.0.0.1:9989\r\n
three Connection: Upgrade\r\n
four Pragma: no-cache\r\n
five Cache-Control: no-cache\r\n
six User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)\r\n
seven Upgrade: websocket\r\n
eight Origin: //xyz.com \r\n
nine Sec-WebSocket-Version: 13\r\n
ten Accept-Encoding: gzip, deflate, br\r\n
eleven Accept-Language: zh-CN,zh; q=0.9,en; q=0.8\r\n
twelve Sec-WebSocket-Key: IqcAWodjyPDJuhGgZwkpKg==\r\n
thirteen Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits\r\n
fourteen \r\n
Requirements for this format are as follows:
Handshake must be a valid HTTP request;
The requested method must be GET, and the HTTP version must be 1.1;
The request must contain the Host field information;
The request must contain the Upgrade field information, and the value must be websocket;
The request must contain the Connection field information, and the value must be Upgrade;
The request must contain the Sec WebSocket Key field, whose value is the client ID encoded in base64 format ；
The request must contain the Sec WebSocket Version field information, and the value must be 13;
The request must contain the Origin field;
The request may contain the Sec WebSocket Protocol field, which specifies the sub protocol;
The request may contain the protocol extension specified in the Sec WebSocket Extensions field;
The request may contain other fields, such as cookies.
After receiving the packet, if the peer supports the WebSocket protocol, it will reply a response in http format. The format of the response message is roughly as follows:
one HTTP/1.1 101 Switching Protocols\r\n
two Upgrade: websocket\r\n
three Connection: Upgrade\r\n
four Sec-WebSocket-Accept: 5wC5L6joP6tl31zpj9OlCNv9Jy4=\r\n
five \r\n
The above lists several fields and corresponding values that must be included in the response message, namely Upgrade 、 Connection 、 Sec-WebSocket-Accept Note: The first line must be HTTP/1.1 101 Switching Protocols\r\n 。
For fields Sec-WebSocket-Accept Field whose value is transmitted from the opposite end Sec-WebSocket-Key The value of is calculated by a certain algorithm, so that both sides of the response can match. The algorithm is as follows:
Splice the Sec-WebSocket Key value with the fixed string "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
The concatenated string is processed by SHA-1, and then the result is encoded by base64.
Algorithm formula:
one mask = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" ; //This is the fixed string used in the algorithm
two accept = base64( sha1( Sec-WebSocket-Key + mask ) );
I implemented the algorithm in C++:
1 namespace uWS {
2
3 struct WebSocketHandshake {
4 template < int N, typename T>
5 struct static_for {
6 void operator () ( uint32_t *a, uint32_t *b) {
7 static_for<N - one , T>()(a, b);
8 T:: template f<N - one >(a, b);
9 }
ten };
eleven
twelve template < typename T>
thirteen struct static_for <0, T> {
fourteen void operator () ( uint32_t *a, uint32_t *hash) {}
fifteen };
sixteen
seventeen template < int state>
eighteen struct Sha1Loop {
nineteen static inline uint32_t rol ( uint32_t value, size_t bits) { return (value << bits) | (value >> ( thirty-two - bits));}
twenty static inline uint32_t blk ( uint32_t b[ sixteen ], size_t i) {
twenty-one return rol(b[(i + thirteen ) & fifteen ] ^ b[(i + eight ) & fifteen ] ^ b[(i + two ) & fifteen ] ^ b[i], one );
twenty-two }
twenty-three
twenty-four template < int i>
twenty-five static inline void f ( uint32_t *a, uint32_t *b) {
twenty-six switch (state) {
twenty-seven case one :
twenty-eight a[i % five ] += ((a[( three + i) % five ] & (a[( two + i) % five ] ^ a[( one + i) % five ])) ^ a[( one + i) % five ]) + b[i] + 0x5a827999 + rol(a[( four + i) % five ], five );
twenty-nine a[( three + i) % five ] = rol(a[( three + i) % five ], thirty );
thirty break ;
thirty-one case two :
thirty-two b[i] = blk(b, i);
thirty-three a[( one + i) % five ] += ((a[( four + i) % five ] & (a[( three + i) % five ] ^ a[( two + i) % five ])) ^ a[( two + i) % five ]) + b[i] + 0x5a827999 + rol(a[( five + i) % five ], five );
thirty-four a[( four + i) % five ] = rol(a[( four + i) % five ], thirty );
thirty-five break ;
thirty-six case three :
thirty-seven b[(i + four ) % sixteen ] = blk(b, (i + four ) % sixteen );
thirty-eight a[i % five ] += (a[( three + i) % five ] ^ a[( two + i) % five ] ^ a[( one + i) % five ]) + b[(i + four ) % sixteen ] + 0x6ed9eba1 + rol(a[( four + i) % five ], five );
thirty-nine a[( three + i) % five ] = rol(a[( three + i) % five ], thirty );
forty break ;
forty-one case four :
forty-two b[(i + eight ) % sixteen ] = blk(b, (i + eight ) % sixteen );
forty-three a[i % five ] += (((a[( three + i) % five ] | a[( two + i) % five ]) & a[( one + i) % five ]) | (a[( three + i) % five ] & a[( two + i) % five ])) + b[(i + eight ) % sixteen ] + 0x8f1bbcdc + rol(a[( four + i) % five ], five );
forty-four a[( three + i) % five ] = rol(a[( three + i) % five ], thirty );
forty-five break ;
forty-six case five :
forty-seven b[(i + twelve ) % sixteen ] = blk(b, (i + twelve ) % sixteen );
forty-eight a[i % five ] += (a[( three + i) % five ] ^ a[( two + i) % five ] ^ a[( one + i) % five ]) + b[(i + twelve ) % sixteen ] + 0xca62c1d6 + rol(a[( four + i) % five ], five );
forty-nine a[( three + i) % five ] = rol(a[( three + i) % five ], thirty );
fifty break ;
fifty-one case six :
fifty-two b[i] += a[ four - i];
fifty-three }
fifty-four }
fifty-five };
fifty-six
fifty-seven /**
fifty-eight *Implementation of sha1 function
fifty-nine */
sixty static inline void sha1 ( uint32_t hash[ five ], uint32_t b[ sixteen ]) {
sixty-one uint32_t a[ five ] = {hash[ four ], hash[ three ], hash[ two ], hash[ one ], hash[ zero ]};
sixty-two static_for< sixteen , Sha1Loop< one >>()(a, b);
sixty-three static_for< four , Sha1Loop< two >>()(a, b);
sixty-four static_for< twenty , Sha1Loop< three >>()(a, b);
sixty-five static_for< twenty , Sha1Loop< four >>()(a, b);
sixty-six static_for< twenty , Sha1Loop< five >>()(a, b);
sixty-seven static_for< five , Sha1Loop< six >>()(a, hash);
sixty-eight }
sixty-nine
seventy /**
seventy-one *Base64 encoding function
seventy-two */
seventy-three static inline void base64 ( unsigned char *src, char *dst) {
seventy-four const char *b64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/" ;
seventy-five for ( int i = zero ; i < eighteen ; i += three ) {
seventy-six *dst++ = b64[(src[i] >> two ) & sixty-three ];
seventy-seven *dst++ = b64[((src[i] & three ) << four ) | ((src[i + one ] & two hundred and forty ) >> four )];
seventy-eight *dst++ = b64[((src[i + one ] & fifteen ) << two ) | ((src[i + two ] & one hundred and ninety-two ) >> six )];
seventy-nine *dst++ = b64[src[i + two ] & sixty-three ];
eighty }
eighty-one *dst++ = b64[(src[ eighteen ] >> two ) & sixty-three ];
eighty-two *dst++ = b64[((src[ eighteen ] & three ) << four ) | ((src[ nineteen ] & two hundred and forty ) >> four )];
eighty-three *dst++ = b64[((src[ nineteen ] & fifteen ) << two )];
eighty-four *dst++ = '=' ;
eighty-five }
eighty-six
eighty-seven public :
eighty-eight /**
eighty-nine *Generate Sec WebSocket Accept algorithm
ninety *@ param input Sec-WebSocket Key value transmitted from the opposite end
ninety-one *@ param output stores the generated Sec WebSocket Accept value
ninety-two */
ninety-three static inline void generate ( const char input[ twenty-four ], char output[ twenty-eight ]) {
ninety-four uint32_t b_output[ five ] = {
ninety-five 0x67452301 , 0xefcdab89 , 0x98badcfe , 0x10325476 , 0xc3d2e1f0
ninety-six };
ninety-seven uint32_t b_input[ sixteen ] = {
ninety-eight zero , zero , zero , zero , zero , zero , 0x32353845 , 0x41464135 , 0x2d453931 , 0x342d3437 , 0x44412d39 ,
ninety-nine 0x3543412d , 0x43354142 , 0x30444338 , 0x35423131 , 0x80000000
one hundred };
one hundred and one
one hundred and two for ( int i = zero ; i < six ; i++) {
one hundred and three b_input[i] = (input[ four * i + three ] & 0xff ) | (input[ four * i + two ] & 0xff ) << eight | (input[ four * i + one ] & 0xff ) << sixteen | (input[ four * i + zero ] & 0xff ) << twenty-four ;
one hundred and four }
one hundred and five sha1(b_output, b_input);
one hundred and six uint32_t last_b[ sixteen ] = { zero , zero , zero , zero , zero , zero , zero , zero , zero , zero , zero , zero , zero , zero , zero , four hundred and eighty };
one hundred and seven sha1(b_output, last_b);
one hundred and eight for ( int i = zero ; i < five ; i++) {
one hundred and nine uint32_t tmp = b_output[i];
one hundred and ten char *bytes = ( char *) &b_output[i];
one hundred and eleven bytes[ three ] = tmp & 0xff ;
one hundred and twelve bytes[ two ] = (tmp >> eight ) & 0xff ;
one hundred and thirteen bytes[ one ] = (tmp >> sixteen ) & 0xff ;
one hundred and fourteen bytes[ zero ] = (tmp >> twenty-four ) & 0xff ;
one hundred and fifteen }
one hundred and sixteen base64(( unsigned char *) b_output, output);
one hundred and seventeen }
one hundred and eighteen };
After the handshake is completed, the communication parties can maintain the connection and send data to each other.
WebSocket protocol format
For RFC documents in WebSocket protocol format, please refer to: [] //tools.ietf.org/html/rfc6455 。
It is often said that the WebSocket protocol is based on the http protocol, so when I first came into contact with the WebSocket protocol, I always thought that every WebSocket packet was in the http format. In fact, in addition to the above mentioned data format used in the handshake process, the WebSocket protocol uses another custom format. Each WebSocket packet is called a Frame, and its format is shown as follows:
Let's introduce the meanings of the fields above one by one:
First byte Content:
FIN Flag, occupying the first bit in the first byte, that is, the highest bit in a byte (a byte is equal to 8 bits). When this flag is set to 0, it means that the current packet has not ended and subsequent fragments of this packet exist; when it is set to 1, it means that the current packet has ended and subsequent fragments of this packet do not exist. When unpacking, if we find that the flag is 1, we need to convert the "package body" data of the current package (that is, in the figure Payload Data ）Cached together with subsequent package fragments is a complete package data.
RSV1 、 RSV2 、 RSV3 One bit for each, three bits in total. These three bits are reserved fields (default is 0). You can use them as some special marks negotiated by the two sides of the communication;
opCode Operation type, four digits in total. The current operation type and its values are as follows:
one // 4 bits
two enum OpCode
three {
four //It means that there will be new frames in the future
five CONTINUATION_FRAME = 0x0 ,
six //The package is a frame of text type
seven TEXT_FRAME = 0x1 ,
eight //The package is a frame of binary type
nine BINARY_FRAME = 0x2 ,
ten //Reserved value
eleven RESERVED1 = 0x3 ,
twelve RESERVED2 = 0x4 ,
thirteen RESERVED3 = 0x5 ,
fourteen RESERVED4 = 0x6 ,
fifteen RESERVED5 = 0x7 ,
sixteen //It is recommended to close the frame at the opposite end
seventeen CLOSE = 0x8 ,
eighteen //Ping Frame in Heartbeat Package
nineteen PING = 0x9 ,
twenty //Pong Frame in heartbeat package
twenty-one PONG = 0xA ,
twenty-two //Reserved value
twenty-three RESERVED6 = 0xB ,
twenty-four RESERVED7 = 0xC ,
twenty-five RESERVED8 = 0xD ,
twenty-six RESERVED9 = 0xE ,
twenty-seven RESERVED10 = 0xF
twenty-eight };
Second byte Content:
mask Flag, occupying one bit. When this flag is 1, it indicates that the frame carries 4 bytes of masking-key Information, not available when it is 0 masking-key Information. masking-key The information is described below.
Payload len , occupying seven digits. This field represents the length information of the package. Because Payload length The value uses the lower seven bits of one byte（ 7 bit ）Therefore, the length range it can represent is 0~127, where one hundred and twenty-six And one hundred and twenty-seven It is used as a special mark.
When the field value is 0~125 When the mask key field is followed by the packet content length; When the value is one hundred and twenty-six Next two The byte content indicates the length of the packet content following the masking key field (i.e Extended Payload Length ）。 Since the maximum unsigned integer represented by 2 bytes is 0xFFFF (The decimal system is 65535, and the compiler provides a macro UINT16_MAX To represent this value). If the packet length exceeds 65535, the packet length cannot be recorded Payload length Set to 127 to use more bytes to represent the packet length.
When Payload length Yes one hundred and twenty-seven Next, use eight The byte content indicates the length of the package content following the masking key field（ Extended Payload Length ）。
To sum up, Payload length=0~125, Extended Payload Length Does not exist, 0 byte; Payload length=126, Extended Payload Length takes 2 bytes; When Payload length=127, Extended Payload Length takes 8 bytes.
In addition, it should be noted that when Payload length=125 or 126, the next two bytes or eight bytes of the actual packet length are stored, and their values must be converted to the network byte order (Big Endian).
Masking-key If the previous mask flag is set to 1, this field exists, accounting for 4 bytes; Otherwise, no storage exists in the Frame masking-key Bytes of the field.
Some materials on the Internet say that, The frame information (packet information) sent by the client (the party that initiatively initiates the handshake request) to the server (the other party that passively accepts the handshake). The mask flag must be 1 The mask flag in the frame information sent by the server to the client is 0. Therefore, there is a 4-byte masking key in the data frame sent from the client to the server, but there is no masking key information in the data frame sent from the server to the client.
I did not see such a mandatory provision in the RFC document of the Websocket protocol. In addition, after studying the implementation of some websocket libraries, I found that this conclusion is not necessarily true, and the data sent by the client may not have the mask flag set.
If present masking-key Information, the data in the data frame (Payload Data in the figure) is all the content after the operation with the masking key. The algorithm is the same whether you get the transmitted data after the original data and the masking key operation, or restore the transmitted data to the original data. The algorithm is as follows:
one Assumptions:
two Original octet-i: the i-th byte of the original data.
three Transformed octet-i: the i-th byte of the converted data.
four j: Is i mod four Results.
five masking- key -Octet-j: mask key The jth byte.
The algorithm is described as: original octet-i and masking key octet-j are XOR to get transformed octet-i.
one j = i MOD four
two transformed-octet-i = original-octet-i XOR masking- key -octet-j
I implemented the algorithm in C++:
one /**
two *@ param src refers to the original data to be transferred before the function is called, and the content after the function is called is masked or unmasked
three *@ param maskingKey four bytes
four */
five void maskAndUnmaskData ( std :: string & src, const char * maskingKey)
six {
seven char j;
eight for ( size_t n = zero ; n < src.length(); ++n)
nine {
ten j = n % four ;
eleven src[n] = src[n] ^ maskingKey[j];
twelve }
thirteen }
The above description may not be clear. Let's take an example. Suppose a client sends a packet to the server, and the mask=1, that is, there is a 4-byte masking key. When the packet body data length is between 0 and 125, the packet structure is:
one Section one Bytes zero Bit=>FIN
two Section one Bytes one ~ three Bit=>RSV1+RSV2+RSV3
three Section one Bytes four ~ seven Bit=>opcode
four Section two Bytes zero Bit=>mask (equal to one )
five Section two Bytes one ~ seven Bit=>inclusion length
six Section three ~ six Bytes=>masking key
seven Section seven Bytes and later=>packet content
In this case, the packet header is 6 bytes in total.
When the packet data length is greater than 125 and less than or equal to UINT16_MAX, the packet structure:
one Section one Bytes zero Bit=>FIN
two Section one Bytes one ~ three Bit=>RSV1+RSV2+RSV3
three Section one Bytes four ~ seven Bit=>opcode
four Section two Bytes zero Bit=>mask (equal to one )
five Section two Bytes one ~ seven Bit=>enable the extended packet header length flag, the value is one hundred and twenty-six
six Section three ~ four Bytes=>packet header length
seven Section five ~ eight Bytes=>masking key
eight Section nine Bytes and later=>packet content
In this case, the packet header is 8 bytes in total.
When the packet data length is greater than UINT16_MAX, the packet structure:
one Section one Bytes zero Bit=>FIN
two Section one Bytes one ~ three Bit=>RSV1+RSV2+RSV3
three Section one Bytes four ~ seven Bit=>opcode
four Section two Bytes zero Bit=>mask (equal to one )
five Section two Bytes one ~ seven Bit=>enable the extended packet header length flag, the value is one hundred and twenty-seven
six Section three ~ ten Bytes=>packet header length
seven Section eleven ~ fourteen Bytes=>masking key
eight Section fifteen Bytes and later=>packet content
In this case, the packet header is 14 bytes in total. Since the length of the storage packet uses 8 bytes for storage (unsigned), the maximum packet length is 0xFFFFFFFFFFFFFFFFFF, which is a very large number. However, in actual development, we cannot use such a long packet, and when the packet exceeds a certain value, we should subcontract (partition).
The logic of subcontracting is also very simple after the previous analysis. Suppose a packet is divided into three pieces, then the first bit FIN of the first byte of the first and second packet pieces should be set to 0, and the OpCode should be set to CONTINUATION_FRAME (also 0); The third package FIN is set to 1, which means that the package is finished at this point, and OpCode is set to the desired type (such as TEXT_FRAME, BINARY_FRAME, etc.). When the peer receives the packet, if the flag FIN=0 or OpCode=0 is found, the data of the packet body will be stored temporarily until the packet with FIN=1 and OpCode ≠ 0 is received, the data of the packet and the previously received data will be put together to form a complete business data. The example code is as follows:
one //After a certain unpacking, the package payloadData is obtained, which is judged according to the FIN flag,
two //If FIN=true, a complete business packet has been received,
three //Call the processPackage() function to process the business data
four //Otherwise, it is temporarily stored in m_strParsedData
five //After processing a complete business package data each time, the data in the staging area m_strParsedData will be cleared
six if (FIN)
seven {
eight m_strParsedData. append (payloadData);
nine processPackage(m_strParsedData);
ten m_strParsedData.clear();
eleven }
twelve else
thirteen {
fourteen m_strParsedData. append (payloadData);
fifteen }
WebSocket compression format
WebSocket also supports compression for packets. Whether compression needs to be enabled or not needs to be negotiated by both communication parties during handshake. Let's take a look at the contents of the package of the initiators when shaking hands:
one GET /realtime HTTP/1.1\r\n
two Host: 127.0.0.1:9989\r\n
three Connection: Upgrade\r\n
four Pragma: no-cache\r\n
five Cache-Control: no-cache\r\n
six User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)\r\n
seven Upgrade: websocket\r\n
eight Origin: //xyz.com \r\n
nine Sec-WebSocket-Version: 13\r\n
ten Accept-Encoding: gzip, deflate, br\r\n
eleven Accept-Language: zh-CN,zh; q=0.9,en; q=0.8\r\n
twelve Sec-WebSocket-Key: IqcAWodjyPDJuhGgZwkpKg==\r\n
thirteen Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits\r\n
fourteen \r\n
In this package Sec-WebSocket-Extensions There is a value in the field permessage-deflate If the initiator supports compression, the packet will be marked with this flag when the handshake is initiated. After the peer receives the packet, if compression is also supported, the reply packet will also be marked with this field. Otherwise, the packet without this flag means that compression is not supported. For example:
one HTTP/1.1 101 Switching Protocols\r\n
two Upgrade: websocket\r\n
three Connection: Upgrade\r\n
four Sec-WebSocket-Accept: 5wC5L6joP6tl31zpj9OlCNv9Jy4=\r\n
five Sec-WebSocket-Extensions: permessage-deflate; client_no_context_takeover
six \r\n
If both parties support compression, the packet body part of the communication packet is compressed, and vice versa. After unpacking the package and getting the package body (i.e. Payload Data), if there is a compression flag when shaking hands and Party B replies that compression is supported, the package body needs to be decompressed; Similarly, when sending data to assemble WebSocket packages, the package body (i.e. Payload Data) needs to be compressed first.
Sample code needs to be decompressed after receiving the package:
one bool MyWebSocketSession::processPackage(const std::string& data )
two {
three std::string out ;
four //M_bClientCompressed determines whether compression is supported during handshake
five if (m_bClientCompressed)
six {
seven //Decompress
eight if (!ZlibUtil::inflate( data , out ))
nine {
ten LOGE( "uncompress failed, dataLength: %d" , data .length());
eleven return false ;
twelve }
thirteen
fourteen }
fifteen else
sixteen out = data ;
seventeen
eighteen //If decompression is not required, out=data; otherwise, out is the decompressed data
nineteen LOGI( "receid data: %s" , out .c_str());
twenty
twenty-one
twenty-two return Process( out );
twenty-three }
Algorithm for compressing packets:
one size_t dataLength = data .length();
two std::string destbuf;
three if (m_bClientCompressed)
four {
five //On-demand compression
six if (!ZlibUtil::deflate( data , destbuf))
seven {
eight LOGE( "compress buf error, data: %s" , data .c_str());
nine return ;
ten }
eleven }
twelve else
thirteen destbuf = data ;
fourteen
fifteen LOGI( "destbuf.length(): %d" , destbuf.length());
The compression and decompression algorithm is the gzip compression algorithm.
Since the maximum number of public account articles is 5000 words, there are 12000 words in the original text of this article, which is omitted when the public account is issued. If you want to get a complete article, please reply to the keyword [websocket protocol analysis] on the public account background. To obtain the complete source code in the article, please reply to the keyword [websocket source code] in the background of the public account.
Finally, due to the limited experience of the author, readers are welcome to put forward valuable suggestions and comments on the views in the article.
