Modern computer systems often include specialized processors that are programmed in domainspecific languages. The compiler-in-the-loop technology, which assumes the joint development of the dedicated processor and compiler, gains in popularity. In this case, the conventional tools (such as GCC and LLVM) are insufficient for the rapid development of optimizing compilers that generate the target code for irregular architectures and static parallelism of operations. In this paper, it is proposed to use methods for solving NP-complete problems for the implementation of machine-dependent compilation phases. These phases are based on the reduction to the SMT problem, which makes it possible to get rid of heuristic and approximate approaches that require complicated software implementation. In particular, it is proposed to implement the synthesis of machine-dependent optimization rules, instruction selection, instruction scheduling, and register allocation using an SMT solver. Practical applications of the developed methods and algorithms are illustrated by the example of a compiler for a specialized processor with an instruction set that accelerates the implementation of lightweight cryptography algorithms on the Internet of Things. The results of compilation and simulation of eight cryptographic primitives for three variants of the specialized processor (CISC-like, VLIW-like and a variant with delayed load instruction) show the practical usefulness of the proposed approach.