Properties for training: SiLU is smooth everywhere, it’s derivative is this is continuous ReLU’s gradient is 0 for every negative input so a unit can get stuck permanently “off”