Do we dream from the second we fall asleep until the second we wake up, or do we only dream during REM sleep?

No matter how we sleep we always dream. When we wake up it will seem like we didn't some times but it is just that we cannot remember the dreams.