Computations in computer graphics, robotics, and probabilistic inference often require differentiating integrals with discontinuous integrands. Popular differentiable programming languages do not support the differentiation of these integrals. We introduce a differentiable programming language, Potto, that is the first to differentiate parametric discontinuities under integration, while supporting first-order functions and compositional evaluation. We extend distribution theory to provide semantic definitions for the derivatives of programs of interest. Using this theory, we present a denotational semantics and a distributional derivative interpretation and show the two accord. We provide operational semantics for both the program and the distributional derivative and soundness theorems.