Xephyr is a nested X server that runs as an X application.
If you wish to run a nested X window, you'll need to specify a display.
$ Xephyr -br -ac -noreset -screen 800x600 :1
This will launch a new Xephyr window with a DISPLAY of ":1". In order to launch an application in that window, you would need to specify that display.
$ DISPLAY=:1 xterm
If you want to launch another WM, spectrwm for example, you would type:
$ DISPLAY=:1 spectrwm
You can also launch Xephyr with your xinitrc using startx:
$ startx -- /usr/bin/Xephyr :1