State-of-the-art optical microscopy fails to image quantitatively complex anisotropic specimens, which are essential in the life sciences, materials science, and optical engineering. In this context, the recent development of Jones-imaging techniques, such as vectorial ptychography, could open up exciting perspectives provided that the measured Jones maps are fully exploited. Here we present a generalized definition of the isotropic phase in the context of arbitrary Jones matrices. We show how to deduce, from combined knowledge of isotropic and anisotropic properties, the underlying structure of the specimen. The power of the method is confirmed by investigations on a biomineral polycrystalline oyster-shell specimen , where we are able to map variations of the three-dimensional orientation of the crystal axis and to highlight structural defects in depth.