An important question for secure all-optical networks (AONs) is how to incorporate security against attacks in the design and engineering of network architectures. In this work, we study the resilience of AON architectures under inband crosstalk attacks. Crosstalk attack propagation depends on both optical devices at the physical layer and wavelength usage at the network layer. This motivates us to employ probabilistic graphical models to model attack propagation. At the physical layer, we use a directed probabilistic graph (Bayesian Belief Network) to model the attack propagation under static network traffic and a given source of attack. At the network layer, we use an undirected probabilistic graph to represent the probability distribution of active connections in the network. The cross-layer model is obtained by combining the physical-and the networklayer models into a factor graph representation. Graphical models provide an explicit representation of interactions between the physical-and the network layer. Furthermore, graphical models facilitate derivations of analytical results on resilience with respect to physical-layer vulnerability, physical topology, and network load. Specifically, we derive bounds on the network resilience for regular topologies. For ring, star, and mesh-torus networks with link-shortest path routing and all-to-all traffic, we show that the average network resilience loss grows linearly with respect to the network load when the network load is small, and polynomially with respect to the probability of attack propagation from node to node along the attacker's route. In addition, numerical results suggest that the sum-product algorithm based on the factor graph representation can be used for computationally efficient evaluation of network resilience for irregular/large topologies.