Modular multilevel converter (MMC) has become the most attractive and promising topology for multi-terminal high-voltage direct current (MTDC) transmission system. Currently, the dq controller and droop controller are widely used in MTDC systems. However, dq control needs phase synchronization by the phase-locked loop (PLL) and ignores the MMC inner dynamics, which jeopardizes the power decoupling and system stability. On the other side, inappropriate droop parameters can cause instability due to the complicated dynamics of MTDC systems. Moreover, the estimation of control parameters stability region will be helpful to guarantee safe operation of the MMC-MTDC systems. In this thesis, a generalized model of the MMC-MTDC systems is developed, which is precise to reflect transient dynamics, and applicable for arbitrary dc network topology and transmission line model. Furthermore, a nonlinear phase-unsynchronized power decoupling control for MMC is proposed. It realizes power decoupling without PLL and MMC output power dynamics are designed as second-order inertial systems for convenient parameter determination. Additionally, a nonlinear droop controller with a reference self-correct algorithm is proposed for improving regulation speed, reducing dc voltage deviation, and maintaining stability. For convenient stability analysis, an inequality-constraint-based method is proposed to efficiently estimate parameter stability regions through constructing the inequality constraints of parameters' variation. To verify the proposed methods, 4-terminal and 14-terminal MMC-MTDC systems are employed. A comparison of dynamic responses between the calculations of nonlinear state-space models in MATLAB and the EMT simulations in PSCAD/EMTDC is conducted to demonstrate the accuracy of the developed model. Then, the proposed phase-unsynchronized power decoupling control is verified by four cases in EMT simulations and four cases in the experimental prototype. Meanwhile, comparisons with the dq control are conducted to demonstrate the benefits of the proposed method. Furthermore, the zero dynamic stability is investigated and the influences of system parameters on stability are analyzed. For the MTDC control, the performance of the proposed nonlinear droop control is validated in the EMT simulations. At last, the effectiveness of the proposed estimation method of parameter stability regions is demonstrated by several examinations including the supremum tests of droop slopes, the stability region sketches on the accuracy, and the unstable operations with predicted improper droop slopes.