Is it too late for me to build up my body's musculature?

I'm 23 and I've never been athletic. I've always been healthy and relatively active, but not athletic, and I've never been much of a muscle-builder or weight lifter. I'm a small guy - 5' 6" and 130 lbs - not particularly muscular. I am wondering if my body is so conditioned at this point to being small and thin that I wouldn't be able to build muscle? I'm not trying to be a bodybuilder here, but I would like to build up my figure a bit - admittedly to compensate for being short. Is my body already too "used" to being thin that it wouldn't build muscle? Would I have to do an exorbitantly greater amount of lifting or weight training or resistance training than most men who are trying to build muscle due to the fact that I've been small for so long?