With the increasing energy-efficiency requirements for the edge computing platforms, the Instruction Set Architecture (ISA) design and precision tuning techniques highlight the possibility of improving the efficiency of dedicated operations. RISC-V as an open-source instruction set shows excellent application potential. On the other hand, the modular ISA design brings elegant design ideas, and designers can pay more attention to the collaborative design of essential ISA and software libraries. To support the design of efficient precision tuned applications, this paper presents some novel architecture and software co-design based on integer and fixed-point computations. Based on a collection of RISC-V instruction custom designs for a fixed-point mathematics library, we propose an optimal decisionmaking methodology for the customized ISA design space exploration to optimize some dedicated algorithms.