anglesey,
If I recall, I think the shot that struck the chimney was grapeshot fired at some distance so that's probably why it didn't do more damage than it did. Had it been a solid cannonball or explosive shell fired from say a Napoleon 12-pounder then it more than likely would have demolished it or at the very least blew gaping holes through it.
Confederate uniforms have always been of interest to me and I have actually done quite a bit of research on the Richmond Clothing Depot as well as sewn quite a few reproduction Richmond Depot shell jackets and trousers over the years. By the Fall of 1862 the Confederacy had established a pretty effective clothing depot system for getting uniforms out to the troops both in the Eastern and Western theaters of the war. While each depot had established a level of uniformity with jacket and trouser styles, they were purchasing cloth (typically a wool-cotton jeans cloth) from various domestic textile mills. The cloth was usually vegetable-dyed with dye recipes often unique to that specific textile mill so by the time it arrived at the depot you had bolts of various shades of "gray". So it was not unusual for a Confederate regiment to be outfitted in the same style shell jacket but in varying shades of color. Also, since the dye was vegetable-based, it would rapidly fade when exposed to the elements and turn from "gray" to various tan and dirty khaki colors (often times why they were referred to as wearing "butternut" uniforms). By 1864, the Richmond Depot had acquired large bolts of "English Army" cloth smuggled through the blockade from Great Britain. This bluish-gray wool kersey was used by the British for making overcoats and trousers for the British military and was of a much higher quality fabric than what was being produced domestically in the South at the time. In addition to the bolts of this cloth, the Irish firm of Peter Tait & Co. out of Limerick, Ireland contracted with the Confederacy to furnish ready-made uniforms made from this same material. By war's end, much of General Robert E. Lee's Army of Northern Virginia had been outfitted with Richmond Depot uniforms and Peter Tait uniforms made from this bluish-gray kersey and ironically presented a more uniform appearance than they had at any other period during the war.