In this paper, we study graphical conditions for structural controllability and accessibility of drifted bilinear systems over Lie groups. We consider a bilinear control system with drift and controlled terms that evolves over the special orthogonal group and the special unitary group. Zero patterns are prescribed for the drift and controlled dynamics with respect to a set of base elements in the corresponding Lie algebra. The drift dynamics must respect a rigid zero-pattern in the sense that the drift takes values as a linear combination of base elements with strictly non-zero coefficients; the controlled dynamics are allowed to follow a free zero pattern with potentially zero coefficients in the configuration of the controlled term by linear combination of the controlled base elements. First of all, for such bilinear systems over the special orthogonal group, the zero patterns are shown to be associated with two undirected graphs whose connectivity and connected components ensure structural controllability. Next, for bilinear systems over the special unitary group, we introduce two edge-colored graphs associated with the drift and controlled zero patterns, and prove structural controllability conditions related to connectivity and the number of edges of a particular color.