What does fastener callout size refer to related to nails/tacks/brads?

The 3/8" designation refers to the length of the tack from under the head to the point. The other designation that you may see is the wire gauge used to make the tack.