While parallel computing greatly improves the computation efficiency of programs, it requires non-trivial knowledge and exercises to rewrite the non-parallel program to execute parallelly. Owing to the complexity of analyzing data dependency, the challenge is yet solved for general programs. In this work, we present a successful machine learning approach to mapping a simple yet important class of programs, tensor computations, to lower-level libraries. Conventionally, the mapping can be simply formulated as a text-to-text task and trained with supervised learning. However, due to the inferred tensor shape change and the flexibility of broadcasting, even powerful models like ChatGPT could not accurately map from a high-level language to lower- level languages. To amend the current neural translation approaches, we propose a hierarchical model to first generate an abstract target program and leverage separate modules to identify the reference and reason for the concrete output literals. We evaluate the proposed method on several tasks, including translation between difference index systems and automatic parallelization. Compared to the neural machine translation-inspired approaches, the proposed hierarchical method achieves much higher accuracy and better generalize to unseen samples