Does it freeze in winter where you live? Ice expands as it forms and this can do damage to masonry. Since your chimney is open to atmosphere, rain water probably evaporates faster than it accumulates. It may be persistently damp in there though, which means mold. Nothing a bleach solution in a spray bottle can’t fix.