Millimeter wave (mmWave) and massive multiple-input multiple-output (MIMO) systems are intrinsic components of 5G and beyond. These systems rely on using beamforming codebooks for both initial access and data transmission. Current beam codebooks, however, are not optimized for the given deployment, which can sometimes incur noticeable performance loss. To address these problems, in this dissertation, three novel machine learning (ML) based frameworks for site-specific analog beam codebook design are proposed. In the first framework, two special neural network-based architectures are designed for learning environment and hardware aware beam codebooks through supervised and self-supervised learning respectively. To avoid explicitly estimating the channels, in the second framework, a deep reinforcement learning-based architecture is developed. The proposed solution significantly relaxes the system requirements and is particularly interesting in scenarios where the channel acquisition is challenging. Building upon it, in the third framework, a sample-efficient online reinforcement learning-based beam codebook design algorithm that learns how to shape the beam patterns to null the interfering directions, without requiring any coordination with the interferers, is developed. In the last part of the dissertation, the proposed beamforming framework is further extended to tackle the beam focusing problem in near field wideband systems. %Specifically, the developed solution can achieve beam focusing without knowing the user position and can account for unknown and non-uniform array geometry. All the frameworks are numerically evaluated and the simulation results highlight their potential of learning site-specific codebooks that adapt to the deployment. Furthermore, a hardware proof-of-concept prototype based on mmWave phased arrays is built and used to evaluate the developed online beam learning solutions in realistic scenarios. The learned beam patterns, measured in an anechoic chamber, show the performance gains of the developed framework. All that highlights a promising ML-based beam/codebook optimization direction for practical and hardware-constrained mmWave and terahertz systems.