goroutine leak: database/sql.(*DB).connectionOpener

problem
solve

problem

we have a long live service, pprof shows that the number of goroutine continues to increase

like this:

...
goroutine 93444 [select, 4266 minutes]:
database/sql.(*DB).connectionOpener(0xc000160270, 0x4cb8780, 0xc0012f4800)
        /usr/local/go/src/database/sql/sql.go:1126 +0xf5
created by database/sql.OpenDB
        /usr/local/go/src/database/sql/sql.go:740 +0x12a

goroutine 204340 [select, 4235 minutes]:
database/sql.(*DB).connectionOpener(0xc001d2af70, 0x4cb8780, 0xc0022b44c0)
        /usr/local/go/src/database/sql/sql.go:1126 +0xf5
created by database/sql.OpenDB
        /usr/local/go/src/database/sql/sql.go:740 +0x12a
...

solve

It's wierd from the first glance, because the call stack doesn't show the caller of sql.OpenDB

So, first step, find OpenDB in sql.go:

func OpenDB(c driver.Connector) *DB {
        ctx, cancel := context.WithCancel(context.Background())
        db := &DB{
                connector:    c,
                openerCh:     make(chan struct{}, connectionRequestQueueSize),
                lastPut:      make(map[*driverConn]string),
                connRequests: make(map[uint64]chan connRequest),
                stop:         cancel,
        }

        go db.connectionOpener(ctx)

        return db
}

go db.connectionOpener(ctx), this line tells us why the call stack looks like that, and we can see DB.stop is cancel func.

Continue to check the function connectionOpener

func (db *DB) connectionOpener(ctx context.Context) {
        for {
                select {
                case <-ctx.Done():
                        return
                case <-db.openerCh:
                        db.openNewConnection(ctx)
                }
        }
}

func (db *DB) Close() error {
        ...
        db.stop()
        return err
}

Notice that db.stop*(which is the cancel func) is called in method *Close, so we know by calling Close, the goroutine leaking problem can be solved.

Now, we know how this problem happens: we open databases periodly, but forgot to close them (or, some of them)

Code snippet:

func newMysql(...) (*mysqlWrapper, error) {
        ...
        DB, err := sql.Open("mysql", dsn)
        if err != nil {
                return nil, err
        }

        err = DB.Ping()
        if err != nil { // BUG HERE!
                return nil, err
        }
        return &mysqlWrapper{DB: DB}, nil
}

func connectionChecker(...) error {
        db, err := newMysql(host, port, user, pass)
        if err != nil {
                return err
        }
        db.Close()
        return nil
}

If DB.Ping fails(eg. wrong password), just returns error, then the openNewConnection goroutine leaks

fix:

...
        err = DB.Ping()
        if err != nil { // BUG HERE!
                DB.Close()
                return nil, err
        }
...

San's notebook

goroutine leak: database/sql.(*DB).connectionOpener

Table of Contents

problem

solve