TCP Port Forwarding Notes

· Systems & tools

Overall Program Structure

Introduction

The program is single-threaded and nonblocking, with client and server components. It maps a TCP service (app) running on a particular port in a private network (app port, such as 80) to a dynamic port on a public server (mapping port, such as 44567). To communicate with the client, the server listens on a port (c-s port, such as 7080). Because of NAT, the client must first send a connect request to the server’s listening port (7080) to establish a connection. This connection is used as masterfd to carry the server’s requests for the client to initiate connections. After receiving the client’s request, the server listens on a dynamic port (such as 44567) as the mapped port for the private-network service, and sends the port information through masterfd to the client. Each time the server subsequently accepts a request, it requests a connection from the client through masterfd. The client then establishes a forwarding tunnel from the app port (such as 80) to the c-s port (such as 7080), while the server establishes a forwarding tunnel from the c-s port to the mapping port. Connecting the two provides port forwarding, with epoll managing multiple such tunnels.

Data Structures

Internally, the client and server do the same thing: for a pair of socket file descriptors, they receive data at one end and forward it to the other. We can therefore encapsulate these operations in an object.

// 对fd的包装
// buf:该fd用来接收数据的缓存区
// iptr, optr记录数据读写位置的指针
// fd_fin, closed,connected:记录fd状态的变量
class fd_buf{
    public:
        int fd;
        char buf[MAXLINE];
        int iptr, optr;
        bool fd_fin,closed,connected;
        // uint32_t mod;
        fd_buf() {
            iptr = optr = 0;
            fd_fin = false;
            connected = false;
            closed = false;
            // mod = EPOLLIN;
        }
        fd_buf(int fd){
            iptr = optr = 0;
            fd_fin = false;
            connected = false;
            closed = false;
            // mod = EPOLLIN;
            this->fd = fd;
        }

        bool is_buf_empty() {
            return iptr == optr;
        }

        bool is_buf_full() {
            return iptr == MAXLINE;
        }
        

};

// 用来完成数据转发的对象
// 从用户的角度看，一段为上行，一段为下载
class tunnel{
    public:
        fd_buf *fd_up, *fd_down;
        double sum_up, sum_down;
        tunnel() {
            sum_up = 0;
            sum_down = 0;
        }
        tunnel(int fd_user, int fd_app) {
            sum_up = 0;
            sum_down = 0;
            fd_up = new fd_buf(fd_user);
            fd_down = new fd_buf(fd_app);
        }
        int do_read(fd_buf* fd_buf_in, fd_buf* pair_fd_buf);

        int do_write(fd_buf* fd_buf_out, fd_buf* pair_fd_buf);

        void fd_user_read(){
            int n = do_read(fd_up, fd_down);
            sum_up += n;
        }

        void fd_app_read(){
            do_read(fd_down, fd_up);
        }

        void fd_user_write(){
            int n = do_write(fd_up, fd_down);
            sum_down += n;
        }

        void fd_app_write(){
            do_write(fd_down, fd_up);
        }
        
        bool is_user_fd(int fd){
            return fd == fd_up->fd;
        }
        ~tunnel() {
            delete fd_down;
            delete fd_up;
        }


};

Nonblocking I/O

Why does blocking occur? For a single connection, later tasks cannot complete correctly until the current stage finishes, so it must block. Why use nonblocking mode? Because we need to manage multiple connections. In a single-threaded program, one blocked connection holds up all the others. One solution is multithreading, with one thread per connection so they do not affect each other. Another is I/O multiplexing: epoll/select monitors sockfd and returns the file descriptors that are ready, preventing one connection from holding up the rest. In socket programming for TCP communication, connect, accept, read, and write can all block. Setting them to nonblocking mode can improve performance and prevent one socket file descriptor from blocking the others.

The setnonblocking Function

// 一个多系统兼容的setnonblocking
int setnonblocking(int fd) {
    if (fd < 0) return -1;
    int flags;
    /* If they have O_NONBLOCK, use the Posix way to do it */
    #if defined(O_NONBLOCK)
        /* Fixme: O_NONBLOCK is defined but broken on SunOS 4.1.x and AIX 3.2.5. */
        if (-1 == (flags = fcntl(fd, F_GETFL, 0)))
        flags = 0;
        return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    #else
        /* Otherwise, use the old way of doing it */
        flags = 1;
        return ioctl(fd, FIOBIO, &flags);
    #endif
}

Nonblocking connect

Why It Blocks

The connect function initiates the TCP three-way handshake by sending a SYN to the server. If the peer does not respond within a certain time, the client retries, potentially for tens of seconds.

Setting Nonblocking Mode

Call setnonblocking(fd) before calling connect.

Handling Nonblocking connect

If fd is set to nonblocking mode before connect is called, an EINPROGRESS error is returned while the TCP connection is still being established. If a connection error occurs, fd becomes both readable and writable; when the connection completes, fd becomes writable. Usually, getsockopt() is used to check for an error and determine whether the connection was established successfully.

 int error;
socklen_t len = sizeof(error);

if(getsockopt(fd,SOL_SOCKET, SO_ERROR, &error, &len) == 0 && error == 0){
        printf("connect success\n");
}
else {
    printf("connect failed\n");
}

Nonblocking accept

Why It Blocks

Normally, after using epoll/select, we do not need to worry about accept blocking, because accept is called only after a connection event is detected. There is one special case, however: a client initiates a connection, epoll/select detects the event and places it in the pending event list, but it is not handled immediately. The client then aborts the connection, and the server removes it. When accept is called, the entire program blocks until a new connection arrives.

How to Handle It

  1. Set the corresponding listenfd to nonblocking mode.
  2. Ignore EWOULDBLOCK, ECONNABORTED, EPROTO, and EINTR errors after accept.

Nonblocking read and write

Conditions for Readability and Writability

Readable

  • Data is available to read.
  • A FIN packet has been received from the peer.
  • A new connection has arrived.
  • An error is pending.

Writable

  • Space is available for writing.
  • An error is pending.

Nonblocking Handling

Call setnonblocking(fd), then use epoll to monitor the corresponding read and write readiness events. After the relevant event is detected, ignore EAGAIN and EWOULDBLOCK errors.

EPOLL Event-Driven Handling

Using tunnel read as an example:

 // 前提 fd&EPOLLIN, 对端fd_peer状态不清楚
 n = read
 if ( n < 0) {
     //由非阻塞引入的错误不需要处理
     if(errno != EAGAIN && errno != EWOULDBLOCK) {
         //先处理在当前条件下可以马上处理的
         //再根据对端状态及应用缓存区状态分类处理

         // 关闭fd后，epoll会自动移除对该fd的监听。
         close(fd)

         //在当前缓存区不为空，对端没有关闭的情况，需要监听对端的可写
         if(!fd_pair.closed && fd.buf not empty ){
             enable fd_peer EPOLLOUT
             //因为fd已经关闭，所以要关闭对端可读
             disable fd EPOLLIN
         }
     }
 }
 else if (n == 0) {
     if(fd_pair.buf not empty ) {
         enable fd EPOLLOUT
     }

     if(!fd_pair.closed && fd.buf is empty) {
         enable fd_pair EPOLLIN
     }

 }
 else {
     if(fd.buf is empty) {
         if(fd_pair not full && !fd_pair.fd_fin) {
             fd_pair EPOLLOUT|EPOLLIN
         }
         else {
             enable fd_pair EPOLLOUT
         }
     }

     if(fd.buf is full) {
         if(fd_pair.buf not empty) {
             enable fd EPOLLOUT
         }
         else{
             disenable fd EPOLLOUT
         }
     }
 }

Handling the Properties of TCP Stream Sockets

TCP guarantees that data reaches the peer in order and without errors, retrying or reporting an error if necessary. However, the number of bytes sent in one operation is not necessarily the number the peer receives in one operation. Keep this in mind when designing an application-layer protocol over TCP.

There are two common solutions. One is fixed-length messages: define a message as a certain number of bytes. On receipt, buffer the bytes and continue receiving from the peer until that length is reached. This approach is relatively inflexible and unsuitable for transmitting complex application-layer data.

The other approach is to design a packet header containing protocol-related information and add a length field when sending. The peer first parses the header to obtain the data length. After receiving a message of that length, it can parse the next header.