In a word, yes. Even when a parent leaves to avoid a dangerous or highly unpleasant situation, if the parent hopes to have physical custody at a later time it's unwise to leave the children behind. The parent who leaves sends a message to the court that the other parent is a suitable choice for physical custody. Also, assuming the children stay in the home where the parents lived as a family, continue in the same school, and participate in their usual activities, a judge may be reluctant to change physical custody, if only to avoid disrupting the children's regular routines.
If a parent must leave the familial home (and wants to be the primary physical custodian), the moving parent should take the children along and, as quickly as possible, file in family court for temporary custody and child support. If this process is delayed, the other parent may go to court first and allege that the kids were taken without that parent's consent or knowledge. Family law judges frown on a parent who removes the children from the home without seeking the court's recognition. A judge may order that the children be returned to the family home, pending future proceedings to determine physical custody.
For more articles and FAQs on child custody and visitation, see Nolo's Child Custody, Child Support & Visitation section.