It is difficult to detect forest fires in complex backgrounds owing to the many interfering factors in forest fire smoke. In this paper, a novel method that combines Time Domain Robust Principal Component Analysis (TRPCA) and a Two-Stream Composed of Visual Geometry Group Network (VGG) and Bi-Long Short-Term Memory (BLSTM) (TSVB) model is proposed for forest fire smoke detection. First, features are extracted from the smoke video from the spatial stream (static) and time stream (dynamic). For the spatial stream, static features are extracted from a single-frame image of the smoke video using the VGG network. For the time stream, continuous-frame binary images of the smoke are obtained using the TRPCA algorithm. Then, the dynamic features of the smoke are extracted by VGG and BLSTM. Finally, the static and dynamic features are fused using a concatenate function to achieve forest fire smoke detection. The experimental results show that compared with the single-feature model, the proposed method effectively improves learning ability and prediction ability, and shows strong robustness against interference factors in a complex background, with accuracy of forest fire smoke detection reaching 90.6%.